Skip to contentSkip to conversation

mm, kvm: allow uffd support in guest_memfd

PATCH Superseded Mike Rapoport 6 participants 40 messages Mar 6 – Mar 30 1 message not in the archive

Subsystems (10)
Memory management VFS KVM Kselftest +6 more subsystems

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Mike Rapoport

author maintainer
#0 lore raw

v2 cover mm, kvm: allow uffd support in guest_memfd

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

These patches enable support for userfaultfd in guest_memfd.

As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.

In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.

Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
The handling of UFFDIO_MISSING in guest_memfd requires ability to remove a folio from page cache, the best way I could find was exporting filemap_remove_folio() to KVM.

I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.

Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.

v1 changes:
* instead of returning uffd-specific values from ->fault() handlers add __do_userfault() helper to resolve user faults in __do_fault()
* address comments from Peter
* rebased on v7.0-c1

RFC: https://lore.kernel.org/all/[email protected]

Mike Rapoport (Microsoft) (11):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_get_pmd() helper.
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic(): remove retry logic

Nikita Kalyazin (3):
KVM: guest_memfd: implement userfaultfd operations
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd

Peter Xu (1):
mm: generalize handling of userfaults in __do_fault()

 include/linux/mm.h                            |   5 +
 include/linux/shmem_fs.h                      |  14 -
 include/linux/userfaultfd_k.h                 |  73 +-
 mm/filemap.c                                  |   1 +
 mm/hugetlb.c                                  |  15 +
 mm/memory.c                                   |  43 ++
 mm/shmem.c                                    | 188 ++---
 mm/userfaultfd.c                              | 692 ++++++++++--------
 .../testing/selftests/kvm/guest_memfd_test.c  | 191 +++++
 virt/kvm/guest_memfd.c                        |  84 ++-
 10 files changed, 858 insertions(+), 448 deletions(-)

base-commit: 6de23f81a5e08be8fbf5e8d7e9febc72a5b5f27f --
2.51.0

author maintainer
#1 lore raw

PATCH v2 1/15 userfaultfd: introduce mfill_copy_folio_locked() helper

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().

This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.

No functional change.

---
 mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
index 927086bb4a3c..32637d557c95 100644
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,
 	return ret;
 }
 
+static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
+{
+	void *kaddr;
+	int ret;
+
+	kaddr = kmap_local_folio(folio, 0);
+	/*
+	 * The read mmap_lock is held here.  Despite the
+	 * mmap_lock being read recursive a deadlock is still
+	 * possible if a writer has taken a lock.  For example:
+	 *
+	 * process A thread 1 takes read lock on own mmap_lock
+	 * process A thread 2 calls mmap, blocks taking write lock
+	 * process B thread 1 takes page fault, read lock on own mmap lock
+	 * process B thread 2 calls mmap, blocks taking write lock
+	 * process A thread 1 blocks taking read lock on process B
+	 * process B thread 1 blocks taking read lock on process A
+	 *
+	 * Disable page faults to prevent potential deadlock
+	 * and retry the copy outside the mmap_lock.
+	 */
+	pagefault_disable();
+	ret = copy_from_user(kaddr, (const void __user *) src_addr,
+			     PAGE_SIZE);
+	pagefault_enable();
+	kunmap_local(kaddr);
+
+	if (ret)
+		return -EFAULT;
+
+	flush_dcache_folio(folio);
+	return ret;
+}
+
 static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 				 struct vm_area_struct *dst_vma,
 				 unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 				 uffd_flags_t flags,
 				 struct folio **foliop)
 {
-	void *kaddr;
 	int ret;
 	struct folio *folio;
 
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 		if (!folio)
 			goto out;
 
-		kaddr = kmap_local_folio(folio, 0);
-		/*
-		 * The read mmap_lock is held here.  Despite the
-		 * mmap_lock being read recursive a deadlock is still
-		 * possible if a writer has taken a lock.  For example:
-		 *
-		 * process A thread 1 takes read lock on own mmap_lock
-		 * process A thread 2 calls mmap, blocks taking write lock
-		 * process B thread 1 takes page fault, read lock on own mmap lock
-		 * process B thread 2 calls mmap, blocks taking write lock
-		 * process A thread 1 blocks taking read lock on process B
-		 * process B thread 1 blocks taking read lock on process A
-		 *
-		 * Disable page faults to prevent potential deadlock
-		 * and retry the copy outside the mmap_lock.
-		 */
-		pagefault_disable();
-		ret = copy_from_user(kaddr, (const void __user *) src_addr,
-				     PAGE_SIZE);
-		pagefault_enable();
-		kunmap_local(kaddr);
+		ret = mfill_copy_folio_locked(folio, src_addr);
 
 		/* fallback to copy_from_user outside mmap_lock */
 		if (unlikely(ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 			/* don't free the page */
 			goto out;
 		}
-
-		flush_dcache_folio(folio);
 	} else {
 		folio = *foliop;
 		*foliop = NULL;
Signature · 1 line
2.51.0
maintainer Reviewed-by on v2 1/15
#2 lore raw

Re: [PATCH v2 01/15] userfaultfd: introduce mfill_copy_folio_locked() helper

Quoted 5 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().

Quoted 8 lines

This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.

No functional change.

---

Signature · 3 lines
Cheers,

David
author maintainer
#3 lore raw

PATCH v2 2/15 userfaultfd: introduce struct mfill_state

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

mfill_atomic() passes a lot of parameters down to its callees.

Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.

Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.

The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.

---
 mm/userfaultfd.c | 148 ++++++++++++++++++++++++++---------------------
 1 file changed, 82 insertions(+), 66 deletions(-)
mm/userfaultfd.c +82 -66
index 32637d557c95..e68d01743b03 100644
@@ -20,6 +20,20 @@
 #include "internal.h"
 #include "swap.h"
 
+struct mfill_state {
+	struct userfaultfd_ctx *ctx;
+	unsigned long src_start;
+	unsigned long dst_start;
+	unsigned long len;
+	uffd_flags_t flags;
+
+	struct vm_area_struct *vma;
+	unsigned long src_addr;
+	unsigned long dst_addr;
+	struct folio *folio;
+	pmd_t *pmd;
+};
+
 static __always_inline
 bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end)
 {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
-				 struct vm_area_struct *dst_vma,
-				 unsigned long dst_addr,
-				 unsigned long src_addr,
-				 uffd_flags_t flags,
-				 struct folio **foliop)
+static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-	int ret;
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
+	unsigned long src_addr = state->src_addr;
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
+	int ret;
 
-	if (!*foliop) {
+	if (!state->folio) {
 		ret = -ENOMEM;
 		folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma,
 					dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 		/* fallback to copy_from_user outside mmap_lock */
 		if (unlikely(ret)) {
 			ret = -ENOENT;
-			*foliop = folio;
+			state->folio = folio;
 			/* don't free the page */
 			goto out;
 		}
 	} else {
-		folio = *foliop;
-		*foliop = NULL;
+		folio = state->folio;
+		state->folio = NULL;
 	}
 
 	/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,
 	return ret;
 }
 
-static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
-				     struct vm_area_struct *dst_vma,
-				     unsigned long dst_addr)
+static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 {
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
+	pmd_t *dst_pmd = state->pmd;
 	pte_t _dst_pte, *dst_pte;
 	spinlock_t *ptl;
 	int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
 }
 
 /* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */
-static int mfill_atomic_pte_continue(pmd_t *dst_pmd,
-				     struct vm_area_struct *dst_vma,
-				     unsigned long dst_addr,
-				     uffd_flags_t flags)
+static int mfill_atomic_pte_continue(struct mfill_state *state)
 {
-	struct inode *inode = file_inode(dst_vma->vm_file);
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
 	pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
+	struct inode *inode = file_inode(dst_vma->vm_file);
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
 	struct page *page;
 	int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd,
 }
 
 /* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
-static int mfill_atomic_pte_poison(pmd_t *dst_pmd,
-				   struct vm_area_struct *dst_vma,
-				   unsigned long dst_addr,
-				   uffd_flags_t flags)
+static int mfill_atomic_pte_poison(struct mfill_state *state)
 {
-	int ret;
+	struct vm_area_struct *dst_vma = state->vma;
 	struct mm_struct *dst_mm = dst_vma->vm_mm;
+	unsigned long dst_addr = state->dst_addr;
+	pmd_t *dst_pmd = state->pmd;
 	pte_t _dst_pte, *dst_pte;
 	spinlock_t *ptl;
+	int ret;
 
 	_dst_pte = make_pte_marker(PTE_MARKER_POISONED);
 	ret = -EAGAIN;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx,
 				    uffd_flags_t flags);
 #endif /* CONFIG_HUGETLB_PAGE */
 
-static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,
-						struct vm_area_struct *dst_vma,
-						unsigned long dst_addr,
-						unsigned long src_addr,
-						uffd_flags_t flags,
-						struct folio **foliop)
+static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 {
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long src_addr = state->src_addr;
+	unsigned long dst_addr = state->dst_addr;
+	struct folio **foliop = &state->folio;
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	ssize_t err;
 
-	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) {
-		return mfill_atomic_pte_continue(dst_pmd, dst_vma,
-						 dst_addr, flags);
-	} else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) {
-		return mfill_atomic_pte_poison(dst_pmd, dst_vma,
-					       dst_addr, flags);
-	}
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+		return mfill_atomic_pte_continue(state);
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
+		return mfill_atomic_pte_poison(state);
 
 	/*
 	 * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,
 	 */
 	if (!(dst_vma->vm_flags & VM_SHARED)) {
 		if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-			err = mfill_atomic_pte_copy(dst_pmd, dst_vma,
-						    dst_addr, src_addr,
-						    flags, foliop);
+			err = mfill_atomic_pte_copy(state);
 		else
-			err = mfill_atomic_pte_zeropage(dst_pmd,
-						 dst_vma, dst_addr);
+			err = mfill_atomic_pte_zeropage(state);
 	} else {
 		err = shmem_mfill_atomic_pte(dst_pmd, dst_vma,
 					     dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 					    unsigned long len,
 					    uffd_flags_t flags)
 {
+	struct mfill_state state = (struct mfill_state){
+		.ctx = ctx,
+		.dst_start = dst_start,
+		.src_start = src_start,
+		.flags = flags,
+
+		.src_addr = src_start,
+		.dst_addr = dst_start,
+	};
 	struct mm_struct *dst_mm = ctx->mm;
 	struct vm_area_struct *dst_vma;
+	long copied = 0;
 	ssize_t err;
 	pmd_t *dst_pmd;
-	unsigned long src_addr, dst_addr;
-	long copied;
-	struct folio *folio;
 
 	/*
 	 * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(src_start + len <= src_start);
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
-	src_addr = src_start;
-	dst_addr = dst_start;
-	copied = 0;
-	folio = NULL;
 retry:
 	/*
 	 * Make sure the vma is not shared, that the dst range is
@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
 		goto out_unlock;
 
-	while (src_addr < src_start + len) {
-		pmd_t dst_pmdval;
+	state.vma = dst_vma;
 
-		VM_WARN_ON_ONCE(dst_addr >= dst_start + len);
+	while (state.src_addr < src_start + len) {
+		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
+
+		pmd_t dst_pmdval;
 
-		dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
 		if (unlikely(!dst_pmd)) {
 			err = -ENOMEM;
 			break;
@@ -827,34 +843,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		 * tables under us; pte_offset_map_lock() will deal with that.
 		 */
 
-		err = mfill_atomic_pte(dst_pmd, dst_vma, dst_addr,
-				       src_addr, flags, &folio);
+		state.pmd = dst_pmd;
+		err = mfill_atomic_pte(&state);
 		cond_resched();
 
 		if (unlikely(err == -ENOENT)) {
 			void *kaddr;
 
 			up_read(&ctx->map_changing_lock);
-			uffd_mfill_unlock(dst_vma);
-			VM_WARN_ON_ONCE(!folio);
+			uffd_mfill_unlock(state.vma);
+			VM_WARN_ON_ONCE(!state.folio);
 
-			kaddr = kmap_local_folio(folio, 0);
+			kaddr = kmap_local_folio(state.folio, 0);
 			err = copy_from_user(kaddr,
-					     (const void __user *) src_addr,
+					     (const void __user *)state.src_addr,
 					     PAGE_SIZE);
 			kunmap_local(kaddr);
 			if (unlikely(err)) {
 				err = -EFAULT;
 				goto out;
 			}
-			flush_dcache_folio(folio);
+			flush_dcache_folio(state.folio);
 			goto retry;
 		} else
-			VM_WARN_ON_ONCE(folio);
+			VM_WARN_ON_ONCE(state.folio);
 
 		if (!err) {
-			dst_addr += PAGE_SIZE;
-			src_addr += PAGE_SIZE;
+			state.dst_addr += PAGE_SIZE;
+			state.src_addr += PAGE_SIZE;
 			copied += PAGE_SIZE;
 
 			if (fatal_signal_pending(current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
 out_unlock:
 	up_read(&ctx->map_changing_lock);
-	uffd_mfill_unlock(dst_vma);
+	uffd_mfill_unlock(state.vma);
 out:
-	if (folio)
-		folio_put(folio);
+	if (state.folio)
+		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);
Signature · 1 line
2.51.0
maintainer Acked-by on v2 2/15
#4 lore raw

Re: [PATCH v2 02/15] userfaultfd: introduce struct mfill_state

Quoted 17 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

mfill_atomic() passes a lot of parameters down to its callees.

Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.

Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.

The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.

---

[...]

Quoted 5 lines
mm/userfaultfd.c +0 -0
 			if (fatal_signal_pending(current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
 out_unlock:
 	up_read(&ctx->map_changing_lock);

mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 891

Quoted 10 lines
mm/userfaultfd.c +3 -3
-	uffd_mfill_unlock(dst_vma);
+	uffd_mfill_unlock(state.vma);
 out:
-	if (folio)
-		folio_put(folio);
+	if (state.folio)
+		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);

I guess, "copied" could be easily avoided by
comparing state.src_addr with start_addr.

Signature · 3 lines
Cheers,

David
author maintainer
#5 lore raw

Re: [PATCH v2 02/15] userfaultfd: introduce struct mfill_state

Quoted 24 lines · David Hildenbrand (Arm)
Quoted 17 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

mfill_atomic() passes a lot of parameters down to its callees.

Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.

Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.

The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.

---

[...]

Quoted 3 lines
mm/userfaultfd.c +0 -0
 			if (fatal_signal_pending(current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
Quoted 16 lines
Quoted 12 lines
mm/userfaultfd.c +3 -3
 out_unlock:
 	up_read(&ctx->map_changing_lock);
-	uffd_mfill_unlock(dst_vma);
+	uffd_mfill_unlock(state.vma);
 out:
-	if (folio)
-		folio_put(folio);
+	if (state.folio)
+		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);
Quoted 1 line
	return copied ? copied : err;

I guess, "copied" could be easily avoided by
comparing state.src_addr with start_addr.

Yeah, but a dedicated variable is more robust against potential changes in state handling. And it's surely more descriptive and shorter than

state.src_addr - src_addr;

Quoted 1 line

Thanks!

Quoted 4 lines
Signature · 3 lines
Cheers,

David
Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#6 lore raw

PATCH v2 3/15 userfaultfd: introduce mfill_get_pmd() helper.

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.

Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().

---
 mm/userfaultfd.c | 103 ++++++++++++++++++++++++-----------------------
 1 file changed, 53 insertions(+), 50 deletions(-)
mm/userfaultfd.c +53 -50
index e68d01743b03..224b55804f99 100644
@@ -157,6 +157,57 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+
+	pgd = pgd_offset(mm, address);
+	p4d = p4d_alloc(mm, pgd, address);
+	if (!p4d)
+		return NULL;
+	pud = pud_alloc(mm, p4d, address);
+	if (!pud)
+		return NULL;
+	/*
+	 * Note that we didn't run this because the pmd was
+	 * missing, the *pmd may be already established and in
+	 * turn it may also be a trans_huge_pmd.
+	 */
+	return pmd_alloc(mm, pud, address);
+}
+
+static int mfill_get_pmd(struct mfill_state *state)
+{
+	struct mm_struct *dst_mm = state->ctx->mm;
+	pmd_t *dst_pmd;
+	pmd_t dst_pmdval;
+
+	dst_pmd = mm_alloc_pmd(dst_mm, state->dst_addr);
+	if (unlikely(!dst_pmd))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	if (unlikely(pmd_none(dst_pmdval)) &&
+	    unlikely(__pte_alloc(dst_mm, dst_pmd)))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	/*
+	 * If the dst_pmd is THP don't override it and just be strict.
+	 * (This includes the case where the PMD used to be THP and
+	 * changed back to none after __pte_alloc().)
+	 */
+	if (unlikely(!pmd_present(dst_pmdval) || pmd_trans_huge(dst_pmdval)))
+		return -EEXIST;
+	if (unlikely(pmd_bad(dst_pmdval)))
+		return -EFAULT;
+
+	state->pmd = dst_pmd;
+	return 0;
+}
+
 /* Check if dst_addr is outside of file's size. Must be called with ptl held. */
 static bool mfill_file_over_size(struct vm_area_struct *dst_vma,
 				 unsigned long dst_addr)
@@ -489,27 +540,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)
 	return ret;
 }
 
-static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
-{
-	pgd_t *pgd;
-	p4d_t *p4d;
-	pud_t *pud;
-
-	pgd = pgd_offset(mm, address);
-	p4d = p4d_alloc(mm, pgd, address);
-	if (!p4d)
-		return NULL;
-	pud = pud_alloc(mm, p4d, address);
-	if (!pud)
-		return NULL;
-	/*
-	 * Note that we didn't run this because the pmd was
-	 * missing, the *pmd may be already established and in
-	 * turn it may also be a trans_huge_pmd.
-	 */
-	return pmd_alloc(mm, pud, address);
-}
-
 #ifdef CONFIG_HUGETLB_PAGE
 /*
  * mfill_atomic processing for HUGETLB vmas.  Note that this routine is
@@ -742,7 +772,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	struct vm_area_struct *dst_vma;
 	long copied = 0;
 	ssize_t err;
-	pmd_t *dst_pmd;
 
 	/*
 	 * Sanitize the command parameters:
@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
-		pmd_t dst_pmdval;
-
-		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
-		if (unlikely(!dst_pmd)) {
-			err = -ENOMEM;
+		err = mfill_get_pmd(&state);
+		if (err)
 			break;
-		}
 
-		dst_pmdval = pmdp_get_lockless(dst_pmd);
-		if (unlikely(pmd_none(dst_pmdval)) &&
-		    unlikely(__pte_alloc(dst_mm, dst_pmd))) {
-			err = -ENOMEM;
-			break;
-		}
-		dst_pmdval = pmdp_get_lockless(dst_pmd);
-		/*
-		 * If the dst_pmd is THP don't override it and just be strict.
-		 * (This includes the case where the PMD used to be THP and
-		 * changed back to none after __pte_alloc().)
-		 */
-		if (unlikely(!pmd_present(dst_pmdval) ||
-				pmd_trans_huge(dst_pmdval))) {
-			err = -EEXIST;
-			break;
-		}
-		if (unlikely(pmd_bad(dst_pmdval))) {
-			err = -EFAULT;
-			break;
-		}
 		/*
 		 * For shmem mappings, khugepaged is allowed to remove page
 		 * tables under us; pte_offset_map_lock() will deal with that.
 		 */
 
-		state.pmd = dst_pmd;
 		err = mfill_atomic_pte(&state);
 		cond_resched();
 
Signature · 1 line
2.51.0
#7 lore raw

Re: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.

Quoted 1 line · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Nit: "." at the end of the patch subject

Quoted 38 lines

There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.

Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().

---
 mm/userfaultfd.c | 103 ++++++++++++++++++++++++-----------------------
 1 file changed, 53 insertions(+), 50 deletions(-)
mm/userfaultfd.c +16 -0
index e68d01743b03..224b55804f99 100644
@@ -157,6 +157,57 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+
+	pgd = pgd_offset(mm, address);
+	p4d = p4d_alloc(mm, pgd, address);
+	if (!p4d)
+		return NULL;
+	pud = pud_alloc(mm, p4d, address);
+	if (!pud)
+		return NULL;
+	/*
+	 * Note that we didn't run this because the pmd was
+	 * missing, the *pmd may be already established and in

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 185

Quoted 10 lines
mm/userfaultfd.c +10 -0
+	 * turn it may also be a trans_huge_pmd.
+	 */
+	return pmd_alloc(mm, pud, address);
+}
+
+static int mfill_get_pmd(struct mfill_state *state)
+{
+	struct mm_struct *dst_mm = state->ctx->mm;
+	pmd_t *dst_pmd;
+	pmd_t dst_pmdval;

I'd just have both on a single line.

Quoted 7 lines
mm/userfaultfd.c +7 -0
+
+	dst_pmd = mm_alloc_pmd(dst_mm, state->dst_addr);
+	if (unlikely(!dst_pmd))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	if (unlikely(pmd_none(dst_pmdval)) &&

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 202

Quoted 10 lines
mm/userfaultfd.c +10 -0
+	    unlikely(__pte_alloc(dst_mm, dst_pmd)))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	/*
+	 * If the dst_pmd is THP don't override it and just be strict.
+	 * (This includes the case where the PMD used to be THP and
+	 * changed back to none after __pte_alloc().)
+	 */
+	if (unlikely(!pmd_present(dst_pmdval) || pmd_trans_huge(dst_pmdval)))

Can we directly switch to pmd_leaf() while touching that?

Quoted 7 lines
mm/userfaultfd.c +7 -0
+		return -EEXIST;
+	if (unlikely(pmd_bad(dst_pmdval)))
+		return -EFAULT;
+
+	state->pmd = dst_pmd;
+	return 0;
+}

[...]

mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 842

Quoted 13 lines
mm/userfaultfd.c +2 -5
 	/*
 	 * Sanitize the command parameters:
@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
-		pmd_t dst_pmdval;
-
-		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
-		if (unlikely(!dst_pmd)) {
-			err = -ENOMEM;
+		err = mfill_get_pmd(&state);
+		if (err)

It's a bit odd that a "get" function doesn't return a PMD pointer but instead stores it in the state.

Maybe more like "mfill_prepare_pmd" ? But actually you want to have a pte table.

mfill_prepare_pte_table() or alternatively mfill_alloc_pte_table() / mfill_alloc_dst_pte_table()

Signature · 3 lines
Cheers,

David
author maintainer
#8 lore raw

Re: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.

Quoted 4 lines · David Hildenbrand (Arm)
Quoted 1 line · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Nit: "." at the end of the patch subject

Oops :/

Quoted 7 lines

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 185

Quoted 5 lines
mm/userfaultfd.c +5 -0
+static int mfill_get_pmd(struct mfill_state *state)
+{
+	struct mm_struct *dst_mm = state->ctx->mm;
+	pmd_t *dst_pmd;
+	pmd_t dst_pmdval;

I'd just have both on a single line.

Can do :)

Quoted 8 lines

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 202

Quoted 6 lines
mm/userfaultfd.c +6 -0
+	/*
+	 * If the dst_pmd is THP don't override it and just be strict.
+	 * (This includes the case where the PMD used to be THP and
+	 * changed back to none after __pte_alloc().)
+	 */
+	if (unlikely(!pmd_present(dst_pmdval) || pmd_trans_huge(dst_pmdval)))

Can we directly switch to pmd_leaf() while touching that?

You mean instead of pmd_trans_huge()?
Yeah, sure.

Quoted 16 lines

mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 842

Quoted 11 lines
mm/userfaultfd.c +2 -5
@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
-		pmd_t dst_pmdval;
-
-		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
-		if (unlikely(!dst_pmd)) {
-			err = -ENOMEM;
+		err = mfill_get_pmd(&state);
+		if (err)

It's a bit odd that a "get" function doesn't return a PMD pointer but instead stores it in the state.

Quoted 5 lines

Maybe more like "mfill_prepare_pmd" ? But actually you want to have a pte table.

mfill_prepare_pte_table() or alternatively mfill_alloc_pte_table() / mfill_alloc_dst_pte_table()

As it actually allocates the pte table once in 512 times, I'd prefer mfill_establish_pmd().

Quoted 3 lines
Signature · 2 lines
Cheers,
David
Signature · 2 lines
Sincerely yours,
Mike.
#9 lore raw

Re: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.

Quoted 6 lines
Quoted 3 lines

mfill_prepare_pte_table() or alternatively mfill_alloc_pte_table() / mfill_alloc_dst_pte_table()

As it actually allocates the pte table once in 512 times, I'd prefer mfill_establish_pmd().

Replying only now (spotted v3 in my inbox)m sounds good to me.

Signature · 3 lines
Cheers,

David
author maintainer
#10 lore raw

PATCH v2 4/15 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.

This function will be used later during refactoring of mfill_atomic_pte_copy().

Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.

---
 mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++-------------------
 1 file changed, 73 insertions(+), 51 deletions(-)
mm/userfaultfd.c +73 -51
index 224b55804f99..baff11e83101 100644
@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static void mfill_put_vma(struct mfill_state *state)
+{
+	up_read(&state->ctx->map_changing_lock);
+	uffd_mfill_unlock(state->vma);
+	state->vma = NULL;
+}
+
+static int mfill_get_vma(struct mfill_state *state)
+{
+	struct userfaultfd_ctx *ctx = state->ctx;
+	uffd_flags_t flags = state->flags;
+	struct vm_area_struct *dst_vma;
+	int err;
+
+	/*
+	 * Make sure the vma is not shared, that the dst range is
+	 * both valid and fully within a single existing vma.
+	 */
+	dst_vma = uffd_mfill_lock(ctx->mm, state->dst_start, state->len);
+	if (IS_ERR(dst_vma))
+		return PTR_ERR(dst_vma);
+
+	/*
+	 * If memory mappings are changing because of non-cooperative
+	 * operation (e.g. mremap) running in parallel, bail out and
+	 * request the user to retry later
+	 */
+	down_read(&ctx->map_changing_lock);
+	err = -EAGAIN;
+	if (atomic_read(&ctx->mmap_changing))
+		goto out_unlock;
+
+	err = -EINVAL;
+
+	/*
+	 * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
+	 * it will overwrite vm_ops, so vma_is_anonymous must return false.
+	 */
+	if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
+	    dst_vma->vm_flags & VM_SHARED))
+		goto out_unlock;
+
+	/*
+	 * validate 'mode' now that we know the dst_vma: don't allow
+	 * a wrprotect copy if the userfaultfd didn't register as WP.
+	 */
+	if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))
+		goto out_unlock;
+
+	if (is_vm_hugetlb_page(dst_vma))
+		goto out;
+
+	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+		goto out_unlock;
+	if (!vma_is_shmem(dst_vma) &&
+	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+		goto out_unlock;
+
+out:
+	state->vma = dst_vma;
+	return 0;
+
+out_unlock:
+	mfill_put_vma(state);
+	return err;
+}
+
 static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
 {
 	pgd_t *pgd;
@@ -768,8 +835,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		.src_addr = src_start,
 		.dst_addr = dst_start,
 	};
-	struct mm_struct *dst_mm = ctx->mm;
-	struct vm_area_struct *dst_vma;
 	long copied = 0;
 	ssize_t err;
 
@@ -784,57 +849,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
 retry:
-	/*
-	 * Make sure the vma is not shared, that the dst range is
-	 * both valid and fully within a single existing vma.
-	 */
-	dst_vma = uffd_mfill_lock(dst_mm, dst_start, len);
-	if (IS_ERR(dst_vma)) {
-		err = PTR_ERR(dst_vma);
+	err = mfill_get_vma(&state);
+	if (err)
 		goto out;
-	}
-
-	/*
-	 * If memory mappings are changing because of non-cooperative
-	 * operation (e.g. mremap) running in parallel, bail out and
-	 * request the user to retry later
-	 */
-	down_read(&ctx->map_changing_lock);
-	err = -EAGAIN;
-	if (atomic_read(&ctx->mmap_changing))
-		goto out_unlock;
-
-	err = -EINVAL;
-	/*
-	 * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
-	 * it will overwrite vm_ops, so vma_is_anonymous must return false.
-	 */
-	if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
-	    dst_vma->vm_flags & VM_SHARED))
-		goto out_unlock;
-
-	/*
-	 * validate 'mode' now that we know the dst_vma: don't allow
-	 * a wrprotect copy if the userfaultfd didn't register as WP.
-	 */
-	if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))
-		goto out_unlock;
 
 	/*
 	 * If this is a HUGETLB vma, pass off to appropriate routine
 	 */
-	if (is_vm_hugetlb_page(dst_vma))
-		return  mfill_atomic_hugetlb(ctx, dst_vma, dst_start,
+	if (is_vm_hugetlb_page(state.vma))
+		return  mfill_atomic_hugetlb(ctx, state.vma, dst_start,
 					     src_start, len, flags);
 
-	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
-		goto out_unlock;
-	if (!vma_is_shmem(dst_vma) &&
-	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
-		goto out_unlock;
-
-	state.vma = dst_vma;
-
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
@@ -853,8 +878,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		if (unlikely(err == -ENOENT)) {
 			void *kaddr;
 
-			up_read(&ctx->map_changing_lock);
-			uffd_mfill_unlock(state.vma);
+			mfill_put_vma(&state);
 			VM_WARN_ON_ONCE(!state.folio);
 
 			kaddr = kmap_local_folio(state.folio, 0);
@@ -883,9 +907,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 			break;
 	}
 
-out_unlock:
-	up_read(&ctx->map_changing_lock);
-	uffd_mfill_unlock(state.vma);
+	mfill_put_vma(&state);
 out:
 	if (state.folio)
 		folio_put(state.folio);
Signature · 1 line
2.51.0

A message that is not in the archive (sent off-list or not yet received): abe1FHyYinvfLYnw@hyeyoo

#12 lore raw

Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

[Adding missing mailing lists to Cc that I omitted by mistake]

Quoted 131 lines · Harry Yoo
Quoted 6 lines · Mike Rapoport (Microsoft)

Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.

This function will be used later during refactoring of mfill_atomic_pte_copy().

Quoted 6 lines

Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.

---

Apparently this patch seems to have two issues... Fortunately, it did not land mainline yet and can be addressed.

Deepanshu and Edward sent fixes, but it should be fixed as part of v3 (if Mike plans to do so) or as a fix-up on patch 4.

Please keep in mind that my understanding of this patchset is limited. I'm just doing some mechanical analysis.

Quoted 22 lines
 mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++-------------------
 1 file changed, 73 insertions(+), 51 deletions(-)
mm/userfaultfd.c +11 -0
index 224b55804f99..baff11e83101 100644
@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static void mfill_put_vma(struct mfill_state *state)
+{
+	up_read(&state->ctx->map_changing_lock);
+	uffd_mfill_unlock(state->vma);
+	state->vma = NULL;
+}
+
+static int mfill_get_vma(struct mfill_state *state)
+{
+	struct userfaultfd_ctx *ctx = state->ctx;
+	uffd_flags_t flags = state->flags;

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 180

Quoted 10 lines
mm/userfaultfd.c +10 -0
+	struct vm_area_struct *dst_vma;
+	int err;
+
+	/*
+	 * Make sure the vma is not shared, that the dst range is
+	 * both valid and fully within a single existing vma.
+	 */
+	dst_vma = uffd_mfill_lock(ctx->mm, state->dst_start, state->len);
+	if (IS_ERR(dst_vma))
+		return PTR_ERR(dst_vma);

state->len is always initialized to zero in patch 2. syzbot triggered a warning in folio_add_new_anon_rmap(), which appears to be due to failing to verify the range in uffd_mfill_lock() (at least syzbot says it's fixed [1]).

[1] https://lore.kernel.org/linux-mm/[email protected]

It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.

At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).

[2] https://lore.kernel.org/linux-mm/[email protected]

Quoted 25 lines
mm/userfaultfd.c +25 -0
+	/*
+	 * If memory mappings are changing because of non-cooperative
+	 * operation (e.g. mremap) running in parallel, bail out and
+	 * request the user to retry later
+	 */
+	down_read(&ctx->map_changing_lock);
+	err = -EAGAIN;
+	if (atomic_read(&ctx->mmap_changing))
+		goto out_unlock;
+
+	err = -EINVAL;
+
+	/*
+	 * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
+	 * it will overwrite vm_ops, so vma_is_anonymous must return false.
+	 */
+	if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
+	    dst_vma->vm_flags & VM_SHARED))
+		goto out_unlock;
+
+	/*
+	 * validate 'mode' now that we know the dst_vma: don't allow
+	 * a wrprotect copy if the userfaultfd didn't register as WP.
+	 */
+	if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))

mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 216

Quoted 10 lines
mm/userfaultfd.c +10 -0
+		goto out_unlock;
+
+	if (is_vm_hugetlb_page(dst_vma))
+		goto out;
+
+	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+		goto out_unlock;
+	if (!vma_is_shmem(dst_vma) &&
+	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+		goto out_unlock;

In case of error, mfill_put_vma() is supposed to unlock appropriate locks but state->vma is not set, so it does nothing.

Pointed out by Edward Adam Davis [3]
[3] https://lore.kernel.org/linux-mm/[email protected]

I think, this should probably be fine in patch 4, but patch 5 adds if (!state->vma) return;, leading to unreleased locks on error paths.

Quoted 13 lines
mm/userfaultfd.c +10 -0
+
+out:
+	state->vma = dst_vma;
+	return 0;
+
+out_unlock:
+	mfill_put_vma(state);
+	return err;
+}
+
 static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
 {
 	pgd_t *pgd;
Signature · 2 lines
Cheers,
Harry / Hyeonggon
#13 lore raw

Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

Quoted 11 lines · Harry Yoo
Quoted 11 lines

It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.

At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).

[2] https://lore.kernel.org/linux-mm/[email protected]

Harry,

You are correct that once vm_refcnt > 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ in vma_start_read(), and CHECK 2 can miss this if mmap_write_unlock() completes before CHECK 2 runs.

Here is the exact race:

vma_start_read():

    /* CHECK 1 */
    if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
        goto err;

    /*
     * RACE WINDOW: vm_refcnt is still 0 here!
     * UFFDIO_UNREGISTER can run:
     *
     *   mmap_write_lock()    -> mm_lock_seq = 11
     *   vma_start_write(vma) -> vm_lock_seq = 11
     *   __split_vma()        -> vma->vm_end = 0x4ca000
     *   mmap_write_unlock()  -> mm_lock_seq = 12
     *
     * writer completes entirely before vm_refcnt++!
     */

    __refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...);
    /* vm_refcnt = 1 now, but vma->vm_end already modified! */

    /* CHECK 2 */
    if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq)))
    /*
     * vm_lock_seq(11) == mm_lock_seq(12)?
     * NO! writer already finished and unlocked!
     * mm_lock_seq incremented to 12 (even=unlocked)
     * CHECK 2 MISSES the race!
     */

    return vma;
    /*
     * returns split vma with vm_end=0x4ca000
     * but vm_refcnt=1 (lock held)
     */

Now mfill_atomic loop runs with split vma:

    while (state.src_addr < src_start + len) {

        /* iteration 1 to N: dst_addr = 0x1b1000 to 0x4c9000
         * all within vma->vm_end(0x4ca000)
         */

        /* iteration N+1: dst_addr = 0x4ca000 */
        err = mfill_atomic_pte(&state);
            mfill_atomic_install_pte(state->vma, dst_addr=0x4ca000)
                folio_add_new_anon_rmap(vma, 0x4ca000)
                    VM_WARN_ON_ONCE(address < vma->vm_start ||
                        address + (nr << 12) > vma->vm_end);
                    /* 0x4ca000 >= vma->vm_end(0x4ca000) -> WARN! */
    }

Without my fix:
CRASH at folio_add_new_anon_rmap

With my fix:

    if (state.dst_addr < state.vma->vm_start ||
        state.dst_addr >= state.vma->vm_end) {
            mfill_put_vma(&state);
            state.dst_start = state.dst_addr;
            state.len = dst_start + len - state.dst_addr;
            err = mfill_get_vma(&state);
            if (err)
                break;
    }
    /* catches split, re-lookups correct VMA safely */

So both fixes are needed:

Harry's fix (state.len):
fixes state.len uninitialized
mfill_get_vma validates correct range
in the first call before loop

My fix (bounds check):
catches split VMA that slipped
through CHECK 2 during loop
because writer finished before
CHECK 2 ran

#14 lore raw

Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

Quoted 34 lines · Deepanshu Kartikey
Quoted 11 lines · Harry Yoo
Quoted 11 lines

It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.

At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).

[2] https://lore.kernel.org/linux-mm/[email protected]

Harry,

You are correct that once vm_refcnt > 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ in vma_start_read(), and CHECK 2 can miss this if mmap_write_unlock() completes before CHECK 2 runs.

Here is the exact race:

vma_start_read():

    /* CHECK 1 */
    if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
        goto err;

    /*
     * RACE WINDOW: vm_refcnt is still 0 here!
     * UFFDIO_UNREGISTER can run:
     *
     *   mmap_write_lock()    -> mm_lock_seq = 11
     *   vma_start_write(vma) -> vm_lock_seq = 11
Quoted 8 lines
     *   __split_vma()        -> vma->vm_end = 0x4ca000
     *   mmap_write_unlock()  -> mm_lock_seq = 12
     *
     * writer completes entirely before vm_refcnt++!
     */

    __refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...);
    /* vm_refcnt = 1 now, but vma->vm_end already modified! */

It is true that vma->vm_end might have changed before acquiring the vma lock, but it doesn't matter as long as you verify the range after acquiring the lock, no? (that's what uffd_mfill_lock() does)

You're not really supposed to read vma->vm_end before acquiring the vma lock and use the value because nothing guarantees that the VMA is stable until the lock is acquired.

Or am I still missing something?

Quoted 13 lines
    /* CHECK 2 */
    if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq)))
    /*
     * vm_lock_seq(11) == mm_lock_seq(12)?
     * NO! writer already finished and unlocked!
     * mm_lock_seq incremented to 12 (even=unlocked)
     * CHECK 2 MISSES the race!
     */
    return vma;
    /*
     * returns split vma with vm_end=0x4ca000
     * but vm_refcnt=1 (lock held)
     */
Signature · 2 lines
Cheers,
Harry / Hyeonggon
#15 lore raw

Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

Quoted 49 lines · Harry Yoo
Quoted 34 lines · Deepanshu Kartikey
Quoted 11 lines · Harry Yoo
Quoted 11 lines

It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.

At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).

[2] https://lore.kernel.org/linux-mm/[email protected]

Harry,

You are correct that once vm_refcnt > 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ in vma_start_read(), and CHECK 2 can miss this if mmap_write_unlock() completes before CHECK 2 runs.

Here is the exact race:

vma_start_read():

    /* CHECK 1 */
    if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
        goto err;

    /*
     * RACE WINDOW: vm_refcnt is still 0 here!
     * UFFDIO_UNREGISTER can run:
     *
     *   mmap_write_lock()    -> mm_lock_seq = 11
     *   vma_start_write(vma) -> vm_lock_seq = 11
Quoted 8 lines
     *   __split_vma()        -> vma->vm_end = 0x4ca000
     *   mmap_write_unlock()  -> mm_lock_seq = 12
     *
     * writer completes entirely before vm_refcnt++!
     */

    __refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...);
    /* vm_refcnt = 1 now, but vma->vm_end already modified! */

It is true that vma->vm_end might have changed before acquiring the vma lock, but it doesn't matter as long as you verify the range after acquiring the lock, no? (that's what uffd_mfill_lock() does)

Quoted 6 lines

You're not really supposed to read vma->vm_end before acquiring the vma lock and use the value because nothing guarantees that the VMA is stable until the lock is acquired.

Or am I still missing something?

Harry, you are right. The real bug is state.len = 0. I withdraw my fix.
Thank you for the explanation

author maintainer
#16 lore raw

PATCH v2 5/15 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).

In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.

Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.

As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.

---
 mm/userfaultfd.c | 78 +++++++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 24 deletions(-)
mm/userfaultfd.c +54 -24
index baff11e83101..828f252c720c 100644
@@ -159,6 +159,9 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 
 static void mfill_put_vma(struct mfill_state *state)
 {
+	if (!state->vma)
+		return;
+
 	up_read(&state->ctx->map_changing_lock);
 	uffd_mfill_unlock(state->vma);
 	state->vma = NULL;
@@ -404,35 +407,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
+static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+{
+	unsigned long src_addr = state->src_addr;
+	void *kaddr;
+	int err;
+
+	/* retry copying with mm_lock dropped */
+	mfill_put_vma(state);
+
+	kaddr = kmap_local_folio(folio, 0);
+	err = copy_from_user(kaddr, (const void __user *) src_addr, PAGE_SIZE);
+	kunmap_local(kaddr);
+	if (unlikely(err))
+		return -EFAULT;
+
+	flush_dcache_folio(folio);
+
+	/* reget VMA and PMD, they could change underneath us */
+	err = mfill_get_vma(state);
+	if (err)
+		return err;
+
+	err = mfill_get_pmd(state);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-	struct vm_area_struct *dst_vma = state->vma;
 	unsigned long dst_addr = state->dst_addr;
 	unsigned long src_addr = state->src_addr;
 	uffd_flags_t flags = state->flags;
-	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
 	int ret;
 
-	if (!state->folio) {
-		ret = -ENOMEM;
-		folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma,
-					dst_addr);
-		if (!folio)
-			goto out;
+	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, state->vma, dst_addr);
+	if (!folio)
+		return -ENOMEM;
 
-		ret = mfill_copy_folio_locked(folio, src_addr);
+	ret = -ENOMEM;
+	if (mem_cgroup_charge(folio, state->vma->vm_mm, GFP_KERNEL))
+		goto out_release;
 
-		/* fallback to copy_from_user outside mmap_lock */
-		if (unlikely(ret)) {
-			ret = -ENOENT;
-			state->folio = folio;
-			/* don't free the page */
-			goto out;
-		}
-	} else {
-		folio = state->folio;
-		state->folio = NULL;
+	ret = mfill_copy_folio_locked(folio, src_addr);
+	if (unlikely(ret)) {
+		/*
+		 * Fallback to copy_from_user outside mmap_lock.
+		 * If retry is successful, mfill_copy_folio_locked() returns
+		 * with locks retaken by mfill_get_vma().
+		 * If there was an error, we must mfill_put_vma() anyway and it
+		 * will take care of unlocking if needed.
+		 */
+		ret = mfill_copy_folio_retry(state, folio);
+		if (ret)
+			goto out_release;
 	}
 
 	/*
@@ -442,17 +473,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
 	 */
 	__folio_mark_uptodate(folio);
 
-	ret = -ENOMEM;
-	if (mem_cgroup_charge(folio, dst_vma->vm_mm, GFP_KERNEL))
-		goto out_release;
-
-	ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
+	ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr,
 				       &folio->page, true, flags);
 	if (ret)
 		goto out_release;
 out:
 	return ret;
 out_release:
+	/* Don't return -ENOENT so that our caller won't retry */
+	if (ret == -ENOENT)
+		ret = -EFAULT;
 	folio_put(folio);
 	goto out;
 }
Signature · 1 line
2.51.0
author maintainer
#17 lore raw

PATCH v2 6/15 userfaultfd: move vma_can_userfault out of line

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

vma_can_userfault() has grown pretty big and it's not called on performance critical path.

Move it out of line.

No functional changes.

---
 include/linux/userfaultfd_k.h | 35 ++---------------------------------
 mm/userfaultfd.c              | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
index fd5f42765497..a49cf750e803 100644
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
 	return vma->vm_flags & __VM_UFFD_FLAGS;
 }
 
-static inline bool vma_can_userfault(struct vm_area_struct *vma,
-				     vm_flags_t vm_flags,
-				     bool wp_async)
-{
-	vm_flags &= __VM_UFFD_FLAGS;
-
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
-	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
-		return false;
-
-	/*
-	 * If wp async enabled, and WP is the only mode enabled, allow any
-	 * memory type.
-	 */
-	if (wp_async && (vm_flags == VM_UFFD_WP))
-		return true;
-
-	/*
-	 * If user requested uffd-wp but not enabled pte markers for
-	 * uffd-wp, then shmem & hugetlbfs are not supported but only
-	 * anonymous.
-	 */
-	if (!uffd_supports_wp_marker() && (vm_flags & VM_UFFD_WP) &&
-	    !vma_is_anonymous(vma))
-		return false;
-
-	/* By default, allow any of anon|shmem|hugetlb */
-	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
-	    vma_is_shmem(vma);
-}
+bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
+		       bool wp_async);
 
 static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
 {
mm/userfaultfd.c +33 -0
index 828f252c720c..c5fd1e5c67b3 100644
@@ -2020,6 +2020,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 	return moved ? moved : err;
 }
 
+bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
+		       bool wp_async)
+{
+	vm_flags &= __VM_UFFD_FLAGS;
+
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+
+	if ((vm_flags & VM_UFFD_MINOR) &&
+	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
+		return false;
+
+	/*
+	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * memory type.
+	 */
+	if (wp_async && (vm_flags == VM_UFFD_WP))
+		return true;
+
+	/*
+	 * If user requested uffd-wp but not enabled pte markers for
+	 * uffd-wp, then shmem & hugetlbfs are not supported but only
+	 * anonymous.
+	 */
+	if (!uffd_supports_wp_marker() && (vm_flags & VM_UFFD_WP) &&
+	    !vma_is_anonymous(vma))
+		return false;
+
+	/* By default, allow any of anon|shmem|hugetlb */
+	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
+	    vma_is_shmem(vma);
+}
+
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
 				     vm_flags_t vm_flags)
 {
Signature · 1 line
2.51.0
author maintainer
#18 lore raw

PATCH v2 7/15 userfaultfd: introduce vm_uffd_ops

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.

First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.

Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.

Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.

Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.

Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.

Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.

---
 include/linux/mm.h            |  5 +++++
 include/linux/userfaultfd_k.h |  6 ++++++
 mm/hugetlb.c                  | 15 +++++++++++++++
 mm/shmem.c                    | 15 +++++++++++++++
 mm/userfaultfd.c              | 36 ++++++++++++++++++++++++++---------
 5 files changed, 68 insertions(+), 9 deletions(-)
include/linux/mm.h +5 -0
index 5be3d8a8f806..b63b28c65676 100644
@@ -741,6 +741,8 @@ struct vm_fault {
 					 */
 };
 
+struct vm_uffd_ops;
+
 /*
  * These are the virtual MM functions - opening of an area, closing and
  * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -826,6 +828,9 @@ struct vm_operations_struct {
 	struct page *(*find_normal_page)(struct vm_area_struct *vma,
 					 unsigned long addr);
 #endif /* CONFIG_FIND_NORMAL_PAGE */
+#ifdef CONFIG_USERFAULTFD
+	const struct vm_uffd_ops *uffd_ops;
+#endif
 };
 
 #ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
index a49cf750e803..56e85ab166c7 100644
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {
 
 extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 
+/* VMA userfaultfd operations */
+struct vm_uffd_ops {
+	/* Checks if a VMA can support userfaultfd */
+	bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+};
+
 /* A combined operation mode + behavior flags. */
 typedef unsigned int __bitwise uffd_flags_t;
 
mm/hugetlb.c +15 -0
index 0beb6e22bc26..077968a8a69a 100644
@@ -4818,6 +4818,18 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)
 	return 0;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static bool hugetlb_can_userfault(struct vm_area_struct *vma,
+				  vm_flags_t vm_flags)
+{
+	return true;
+}
+
+static const struct vm_uffd_ops hugetlb_uffd_ops = {
+	.can_userfault = hugetlb_can_userfault,
+};
+#endif
+
 /*
  * When a new function is introduced to vm_operations_struct and added
  * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4831,6 +4843,9 @@ const struct vm_operations_struct hugetlb_vm_ops = {
 	.close = hugetlb_vm_op_close,
 	.may_split = hugetlb_vm_op_split,
 	.pagesize = hugetlb_vm_op_pagesize,
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops = &hugetlb_uffd_ops,
+#endif
 };
 
 static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +15 -0
index b40f3cd48961..f2a25805b9bf 100644
@@ -3294,6 +3294,15 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
 	shmem_inode_unacct_blocks(inode, 1);
 	return ret;
 }
+
+static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	return true;
+}
+
+static const struct vm_uffd_ops shmem_uffd_ops = {
+	.can_userfault	= shmem_can_userfault,
+};
 #endif /* CONFIG_USERFAULTFD */
 
 #ifdef CONFIG_TMPFS
@@ -5313,6 +5322,9 @@ static const struct vm_operations_struct shmem_vm_ops = {
 	.set_policy     = shmem_set_policy,
 	.get_policy     = shmem_get_policy,
 #endif
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops	= &shmem_uffd_ops,
+#endif
 };
 
 static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5322,6 +5334,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = {
 	.set_policy     = shmem_set_policy,
 	.get_policy     = shmem_get_policy,
 #endif
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops	= &shmem_uffd_ops,
+#endif
 };
 
 int shmem_init_fs_context(struct fs_context *fc)
mm/userfaultfd.c +27 -9
index c5fd1e5c67b3..b55d4a8d88cc 100644
@@ -34,6 +34,25 @@ struct mfill_state {
 	pmd_t *pmd;
 };
 
+static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	/* anonymous memory does not support MINOR mode */
+	if (vm_flags & VM_UFFD_MINOR)
+		return false;
+	return true;
+}
+
+static const struct vm_uffd_ops anon_uffd_ops = {
+	.can_userfault	= anon_can_userfault,
+};
+
+static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma)
+{
+	if (vma_is_anonymous(vma))
+		return &anon_uffd_ops;
+	return vma->vm_ops ? vma->vm_ops->uffd_ops : NULL;
+}
+
 static __always_inline
 bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end)
 {
@@ -2023,13 +2042,15 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 		       bool wp_async)
 {
-	vm_flags &= __VM_UFFD_FLAGS;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
-	if (vma->vm_flags & VM_DROPPABLE)
+	/* only VMAs that implement vm_uffd_ops are supported */
+	if (!ops)
 		return false;
 
-	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
+	vm_flags &= __VM_UFFD_FLAGS;
+
+	if (vma->vm_flags & VM_DROPPABLE)
 		return false;
 
 	/*
@@ -2041,16 +2062,13 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 
 	/*
 	 * If user requested uffd-wp but not enabled pte markers for
-	 * uffd-wp, then shmem & hugetlbfs are not supported but only
-	 * anonymous.
+	 * uffd-wp, then only anonymous memory is supported
 	 */
 	if (!uffd_supports_wp_marker() && (vm_flags & VM_UFFD_WP) &&
 	    !vma_is_anonymous(vma))
 		return false;
 
-	/* By default, allow any of anon|shmem|hugetlb */
-	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
-	    vma_is_shmem(vma);
+	return ops->can_userfault(vma, vm_flags);
 }
 
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
Signature · 1 line
2.51.0
author maintainer
#19 lore raw

Re: [PATCH v2 07/15] userfaultfd: introduce vm_uffd_ops

Review

Quoted 10 lines · Mike Rapoport

bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,

(quoted diff) +3 -2
 		       bool wp_async)
 {
-	vm_flags &= __VM_UFFD_FLAGS;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
-	if (vma->vm_flags & VM_DROPPABLE)
+	/* only VMAs that implement vm_uffd_ops are supported */
+	if (!ops)
 		return false;

Just found out that rejecting a VMA that does not have uffd_ops but was registered in WP-only mode with WP_ASYNC uffd context breaks pagemap_ioctl() test and more broadly it breaks tracking of writes in SysV shared memory areas.

This is weird that it's possible to use uffd with SysV SHM, but it's out there for some time and I afraid we can't change that :/

Andrew, can you apply this as a fixup please

From 6e3319ceab93d84558e735e1f4f451e80c85b267 Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (Microsoft)" <[email protected]>
Date: Wed, 11 Mar 2026 20:21:38 +0200
Subject: [PATCH 1/1] userfaultfd: allow registration of WP_ASYNC for any VMA

Registration of a VMA with WP_ASYNC userfaulfd context in write-protect mode does not require any VMA-specific resolution of user faults and these faults are completely handled by the generic page fault handler.

This functionality existed since the introduction of WP_ASYNC mode and it allows tracking writes to SysV shared memory mappings (shmget(2) and shmat(2)).

Move the check for WP mode before checking for presence of ->uffd_ops in a VMA to restore the original behaviour.

---
 mm/userfaultfd.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
mm/userfaultfd.c +8 -8
index b55d4a8d88cc..436795bf218e 100644
@@ -2044,22 +2044,22 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 {
 	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
-	/* only VMAs that implement vm_uffd_ops are supported */
-	if (!ops)
-		return false;
-
 	vm_flags &= __VM_UFFD_FLAGS;
 
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
 	/*
-	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * If WP is the only mode enabled and context is wp async, allow any
 	 * memory type.
 	 */
 	if (wp_async && (vm_flags == VM_UFFD_WP))
 		return true;
 
+	/* For any other mode reject VMAs that don't implement vm_uffd_ops */
+	if (!ops)
+		return false;
+
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+
 	/*
 	 * If user requested uffd-wp but not enabled pte markers for
 	 * uffd-wp, then only anonymous memory is supported
Signature · 1 line
2.51.0
author maintainer
#20 lore raw

PATCH v2 8/15 shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.

Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.

---
 include/linux/userfaultfd_k.h |  7 +++++++
 mm/shmem.c                    | 15 ++++++++++++++-
 mm/userfaultfd.c              | 32 ++++++++++++++++----------------
 3 files changed, 37 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
index 56e85ab166c7..66dfc3c164e6 100644
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 struct vm_uffd_ops {
 	/* Checks if a VMA can support userfaultfd */
 	bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+	/*
+	 * Called to resolve UFFDIO_CONTINUE request.
+	 * Should return the folio found at pgoff in the VMA's pagecache if it
+	 * exists or ERR_PTR otherwise.
+	 * The returned folio is locked and with reference held.
+	 */
+	struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
 };
 
 /* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
index f2a25805b9bf..7bd887b64f62 100644
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
 	return ret;
 }
 
+static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+	struct folio *folio;
+	int err;
+
+	err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+	if (err)
+		return ERR_PTR(err);
+
+	return folio;
+}
+
 static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
 {
 	return true;
 }
 
 static const struct vm_uffd_ops shmem_uffd_ops = {
-	.can_userfault	= shmem_can_userfault,
+	.can_userfault		= shmem_can_userfault,
+	.get_folio_noalloc	= shmem_get_folio_noalloc,
 };
 #endif /* CONFIG_USERFAULTFD */
 
mm/userfaultfd.c +16 -16
index b55d4a8d88cc..98ade14eaa5b 100644
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)
 	struct userfaultfd_ctx *ctx = state->ctx;
 	uffd_flags_t flags = state->flags;
 	struct vm_area_struct *dst_vma;
+	const struct vm_uffd_ops *ops;
 	int err;
 
 	/*
@@ -231,10 +232,12 @@ static int mfill_get_vma(struct mfill_state *state)
 	if (is_vm_hugetlb_page(dst_vma))
 		goto out;
 
-	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+	ops = vma_uffd_ops(dst_vma);
+	if (!ops)
 		goto out_unlock;
-	if (!vma_is_shmem(dst_vma) &&
-	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE) &&
+	    !ops->get_folio_noalloc)
 		goto out_unlock;
 
 out:
@@ -577,6 +580,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 static int mfill_atomic_pte_continue(struct mfill_state *state)
 {
 	struct vm_area_struct *dst_vma = state->vma;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(dst_vma);
 	unsigned long dst_addr = state->dst_addr;
 	pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
 	struct inode *inode = file_inode(dst_vma->vm_file);
@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
 	struct page *page;
 	int ret;
 
-	ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+	if (!ops)
+		return -EOPNOTSUPP;
+
+	folio = ops->get_folio_noalloc(inode, pgoff);
 	/* Our caller expects us to return -EFAULT if we failed to find folio */
-	if (ret == -ENOENT)
-		ret = -EFAULT;
-	if (ret)
-		goto out;
-	if (!folio) {
-		ret = -EFAULT;
-		goto out;
-	}
+	if (IS_ERR_OR_NULL(folio))
+		return -EFAULT;
 
 	page = folio_file_page(folio, pgoff);
 	if (PageHWPoison(page)) {
@@ -609,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
 		goto out_release;
 
 	folio_unlock(folio);
-	ret = 0;
-out:
-	return ret;
+	return 0;
+
 out_release:
 	folio_unlock(folio);
 	folio_put(folio);
-	goto out;
+	return ret;
 }
 
 /* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.51.0
Reviewed-by on v2 8/15
#21 lore raw

Re: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

Quoted 11 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.

Quoted 5 lines

Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.

I found myself asking why HugeTLB wasn't included here; it's because we call into hugetlb_mfill_atomic_pte() pretty early. It would be nice to have HugeTLB implement get_folio_noalloc() and use the same path as shmem, but we don't have to do that now.

Feel free to add:

Quoted 93 lines
---
 include/linux/userfaultfd_k.h |  7 +++++++
 mm/shmem.c                    | 15 ++++++++++++++-
 mm/userfaultfd.c              | 32 ++++++++++++++++----------------
 3 files changed, 37 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
index 56e85ab166c7..66dfc3c164e6 100644
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 struct vm_uffd_ops {
        /* Checks if a VMA can support userfaultfd */
        bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+       /*
+        * Called to resolve UFFDIO_CONTINUE request.
+        * Should return the folio found at pgoff in the VMA's pagecache if it
+        * exists or ERR_PTR otherwise.
+        * The returned folio is locked and with reference held.
+        */
+       struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
 };
 
 /* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
index f2a25805b9bf..7bd887b64f62 100644
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
        return ret;
 }
 
+static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+       struct folio *folio;
+       int err;
+
+       err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+       if (err)
+               return ERR_PTR(err);
+
+       return folio;
+}
+
 static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
 {
        return true;
 }
 
 static const struct vm_uffd_ops shmem_uffd_ops = {
-       .can_userfault  = shmem_can_userfault,
+       .can_userfault          = shmem_can_userfault,
+       .get_folio_noalloc      = shmem_get_folio_noalloc,
 };
 #endif /* CONFIG_USERFAULTFD */
 
mm/userfaultfd.c +7 -3
index b55d4a8d88cc..98ade14eaa5b 100644
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)
        struct userfaultfd_ctx *ctx = state->ctx;
        uffd_flags_t flags = state->flags;
        struct vm_area_struct *dst_vma;
+       const struct vm_uffd_ops *ops;
        int err;
 
        /*
@@ -231,10 +232,12 @@ static int mfill_get_vma(struct mfill_state *state)
        if (is_vm_hugetlb_page(dst_vma))
                goto out;
 
-       if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+       ops = vma_uffd_ops(dst_vma);
+       if (!ops)
                goto out_unlock;
-       if (!vma_is_shmem(dst_vma) &&
-           uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE) &&
+           !ops->get_folio_noalloc)
                goto out_unlock;
 
 out:
@@ -577,6 +580,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 static int mfill_atomic_pte_continue(struct mfill_state *state)
 {
        struct vm_area_struct *dst_vma = state->vma;
+       const struct vm_uffd_ops *ops = vma_uffd_ops(dst_vma);
        unsigned long dst_addr = state->dst_addr;
        pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
        struct inode *inode = file_inode(dst_vma->vm_file);

mm/userfaultfd.c static int mfill_atomic_pte_continue(struct mfill_state *state) · line 594

Quoted 6 lines
mm/userfaultfd.c +2 -1
@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
        struct page *page;
        int ret;
 
-       ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+       if (!ops)
+               return -EOPNOTSUPP;

Perhaps WARN_ON_ONCE(!ops)?

Quoted 36 lines
mm/userfaultfd.c +7 -13
+
+       folio = ops->get_folio_noalloc(inode, pgoff);
        /* Our caller expects us to return -EFAULT if we failed to find folio */
-       if (ret == -ENOENT)
-               ret = -EFAULT;
-       if (ret)
-               goto out;
-       if (!folio) {
-               ret = -EFAULT;
-               goto out;
-       }
+       if (IS_ERR_OR_NULL(folio))
+               return -EFAULT;
 
        page = folio_file_page(folio, pgoff);
        if (PageHWPoison(page)) {
@@ -609,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
                goto out_release;
 
        folio_unlock(folio);
-       ret = 0;
-out:
-       return ret;
+       return 0;
+
 out_release:
        folio_unlock(folio);
        folio_put(folio);
-       goto out;
+       return ret;
 }
 
 /* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
--

2.51.0

maintainer
#22 lore raw

Re: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

Quoted 18 lines · On Thu, 26 Mar 2026 16:43:27 -0700 James Houghton <[email protected]> wrote:
Quoted 11 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.

Quoted 5 lines

Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.

Quoted 8 lines

I found myself asking why HugeTLB wasn't included here; it's because we call into hugetlb_mfill_atomic_pte() pretty early. It would be nice to have HugeTLB implement get_folio_noalloc() and use the same path as shmem, but we don't have to do that now.

Feel free to add:

Thanks.

Quoted 11 lines
Quoted 11 lines

static int mfill_atomic_pte_continue(struct mfill_state *state)

 {
        struct vm_area_struct *dst_vma = state->vma;
mm/userfaultfd.c +1 -0
+       const struct vm_uffd_ops *ops = vma_uffd_ops(dst_vma);
        unsigned long dst_addr = state->dst_addr;
        pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
        struct inode *inode = file_inode(dst_vma->vm_file);
@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
        struct page *page;
        int ret;
 
Quoted 6 lines

mm/userfaultfd.c static int mfill_atomic_pte_continue(struct mfill_state *state) · line 594

Quoted 3 lines
mm/userfaultfd.c +2 -1
-       ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+       if (!ops)
+               return -EOPNOTSUPP;

Perhaps WARN_ON_ONCE(!ops)?

fyi, I (and I expect many others) tend to consider a "Reviewed-by:" as end-of-message, So if you have additional comments then add something like "comments below" so we know to keep scrolling!

If there are no additional comments, simply zap all quoted material after the R-b to remove all doubt.

author maintainer
#23 lore raw

Re: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

In reply to James Houghton

Hi James,

Quoted 16 lines · James Houghton
Quoted 15 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.

Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.

Quoted 6 lines
Quoted 1 line

I found myself asking why HugeTLB wasn't included here; it's because we call into hugetlb_mfill_atomic_pte() pretty early. It would be nice to have HugeTLB implement get_folio_noalloc() and use the same path as shmem, but we don't have to do that now.

You are welcome to join the effort ;-)

Quoted 3 lines

Feel free to add:

Thanks!

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#24 lore raw

PATCH v2 9/15 userfaultfd: introduce vm_uffd_ops->alloc_folio()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

---
 include/linux/userfaultfd_k.h |  6 +++
 mm/userfaultfd.c              | 92 ++++++++++++++++++-----------------
 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
index 66dfc3c164e6..4d8b879eed91 100644
@@ -91,6 +91,12 @@ struct vm_uffd_ops {
 	 * The returned folio is locked and with reference held.
 	 */
 	struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
+	/*
+	 * Called during resolution of UFFDIO_COPY request.
+	 * Should return allocate a and return folio or NULL if allocation fails.
+	 */
+	struct folio *(*alloc_folio)(struct vm_area_struct *vma,
+				     unsigned long addr);
 };
 
 /* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44

142 lines not on this page · raw message

Signature · 1 line
2.51.0
Reviewed-by on v2 9/15
#25 lore raw

Re: [PATCH v2 09/15] userfaultfd: introduce vm_uffd_ops->alloc_folio()

Quoted 15 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Quoted 7 lines

Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

Feel free to add:

Quoted 10 lines
---
 include/linux/userfaultfd_k.h |  6 +++
 mm/userfaultfd.c              | 92 ++++++++++++++++++-----------------
 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +0 -0
index 66dfc3c164e6..4d8b879eed91 100644

include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 96

Quoted 6 lines
include/linux/userfaultfd_k.h +3 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {
         * The returned folio is locked and with reference held.
         */
        struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
+       /*
+        * Called during resolution of UFFDIO_COPY request.
+        * Should return allocate a and return folio or NULL if allocation fails.

"Should allocate and return a folio or NULL if allocation fails."

I see this mistake is fixed in the next patch. :)

Quoted 73 lines
include/linux/userfaultfd_k.h +3 -0
+        */
+       struct folio *(*alloc_folio)(struct vm_area_struct *vma,
+                                    unsigned long addr);
 };
 
 /* A combined operation mode + behavior flags. */
mm/userfaultfd.c +23 -10
index 98ade14eaa5b..31f3ab6a73e2 100644
@@ -42,8 +42,26 @@ static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
        return true;
 }
 
+static struct folio *anon_alloc_folio(struct vm_area_struct *vma,
+                                     unsigned long addr)
+{
+       struct folio *folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma,
+                                             addr);
+
+       if (!folio)
+               return NULL;
+
+       if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) {
+               folio_put(folio);
+               return NULL;
+       }
+
+       return folio;
+}
+
 static const struct vm_uffd_ops anon_uffd_ops = {
        .can_userfault  = anon_can_userfault,
+       .alloc_folio    = anon_alloc_folio,
 };
 
 static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma)
@@ -458,7 +476,8 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
        return 0;
 }
 
-static int mfill_atomic_pte_copy(struct mfill_state *state)
+static int __mfill_atomic_pte(struct mfill_state *state,
+                             const struct vm_uffd_ops *ops)
 {
        unsigned long dst_addr = state->dst_addr;
        unsigned long src_addr = state->src_addr;
@@ -466,16 +485,12 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
        struct folio *folio;
        int ret;
 
-       folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, state->vma, dst_addr);
+       folio = ops->alloc_folio(state->vma, state->dst_addr);
        if (!folio)
                return -ENOMEM;
 
-       ret = -ENOMEM;
-       if (mem_cgroup_charge(folio, state->vma->vm_mm, GFP_KERNEL))
-               goto out_release;
-
-       ret = mfill_copy_folio_locked(folio, src_addr);
-       if (unlikely(ret)) {
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY)) {
+               ret = mfill_copy_folio_locked(folio, src_addr);
                /*
                 * Fallback to copy_from_user outside mmap_lock.
                 * If retry is successful, mfill_copy_folio_locked() returns
@@ -483,9 +498,15 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
                 * If there was an error, we must mfill_put_vma() anyway and it
                 * will take care of unlocking if needed.
                 */
-               ret = mfill_copy_folio_retry(state, folio);
-               if (ret)

mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 509

Quoted 10 lines
mm/userfaultfd.c +9 -1
-                       goto out_release;
+               if (unlikely(ret)) {
+                       ret = mfill_copy_folio_retry(state, folio);
+                       if (ret)
+                               goto err_folio_put;
+               }
+       } else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) {
+               clear_user_highpage(&folio->page, state->dst_addr);
+       } else {
+               VM_WARN_ONCE(1, "unknown UFFDIO operation");

"Unknown UFFDIO operation. flags=%x", flags

seems a little better to me.

Quoted 78 lines
mm/userfaultfd.c +16 -34
        }
 
        /*
@@ -498,47 +519,30 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
        ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr,
                                       &folio->page, true, flags);
        if (ret)
-               goto out_release;
-out:
-       return ret;
-out_release:
+               goto err_folio_put;
+
+       return 0;
+
+err_folio_put:
+       folio_put(folio);
        /* Don't return -ENOENT so that our caller won't retry */
        if (ret == -ENOENT)
                ret = -EFAULT;
-       folio_put(folio);
-       goto out;
+       return ret;
 }
 
-static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,
-                                        struct vm_area_struct *dst_vma,
-                                        unsigned long dst_addr)
+static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-       struct folio *folio;
-       int ret = -ENOMEM;
-
-       folio = vma_alloc_zeroed_movable_folio(dst_vma, dst_addr);
-       if (!folio)
-               return ret;
-
-       if (mem_cgroup_charge(folio, dst_vma->vm_mm, GFP_KERNEL))
-               goto out_put;
+       const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
 
-       /*
-        * The memory barrier inside __folio_mark_uptodate makes sure that
-        * zeroing out the folio become visible before mapping the page
-        * using set_pte_at(). See do_anonymous_page().
-        */
-       __folio_mark_uptodate(folio);
+       return __mfill_atomic_pte(state, ops);
+}
 
-       ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
-                                      &folio->page, true, 0);
-       if (ret)
-               goto out_put;
+static int mfill_atomic_pte_zeroed_folio(struct mfill_state *state)
+{
+       const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
 
-       return 0;
-out_put:
-       folio_put(folio);
-       return ret;
+       return __mfill_atomic_pte(state, ops);
 }
 
 static int mfill_atomic_pte_zeropage(struct mfill_state *state)
@@ -551,7 +555,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
        int ret;
 
        if (mm_forbids_zeropage(dst_vma->vm_mm))
-               return mfill_atomic_pte_zeroed_folio(dst_pmd, dst_vma, dst_addr);
+               return mfill_atomic_pte_zeroed_folio(state);
 
        _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr),
                                         dst_vma->vm_page_prot));
--

2.51.0

author maintainer
#26 lore raw

Re: [PATCH v2 09/15] userfaultfd: introduce vm_uffd_ops->alloc_folio()

Quoted 19 lines · James Houghton
Quoted 18 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Quoted 8 lines
Quoted 4 lines

Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

Feel free to add:

Thanks!

Quoted 19 lines
Quoted 10 lines
---
 include/linux/userfaultfd_k.h |  6 +++
 mm/userfaultfd.c              | 92 ++++++++++++++++++-----------------
 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +0 -0
index 66dfc3c164e6..4d8b879eed91 100644

include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 96

Quoted 6 lines
include/linux/userfaultfd_k.h +3 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {
         * The returned folio is locked and with reference held.
         */
        struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
+       /*
+        * Called during resolution of UFFDIO_COPY request.
+        * Should return allocate a and return folio or NULL if allocation fails.

"Should allocate and return a folio or NULL if allocation fails."

Quoted 1 line

I see this mistake is fixed in the next patch. :)

Endless rebases :)
Will try to sort it out this time :)

Quoted 19 lines
Quoted 6 lines
include/linux/userfaultfd_k.h +0 -2
@@ -483,9 +498,15 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
                 * If there was an error, we must mfill_put_vma() anyway and it
                 * will take care of unlocking if needed.
                 */
-               ret = mfill_copy_folio_retry(state, folio);
-               if (ret)

include/linux/userfaultfd_k.h static int mfill_atomic_pte_copy(struct mfill_state *state) · line 509

Quoted 10 lines
include/linux/userfaultfd_k.h +9 -1
-                       goto out_release;
+               if (unlikely(ret)) {
+                       ret = mfill_copy_folio_retry(state, folio);
+                       if (ret)
+                               goto err_folio_put;
+               }
+       } else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) {
+               clear_user_highpage(&folio->page, state->dst_addr);
+       } else {
+               VM_WARN_ONCE(1, "unknown UFFDIO operation");

"Unknown UFFDIO operation. flags=%x", flags

Quoted 1 line

seems a little better to me.

Yeah, why not.

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#27 lore raw

PATCH v2 10/15 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.

---
 include/linux/shmem_fs.h      |  14 ----
 include/linux/userfaultfd_k.h |  21 +++--
 mm/shmem.c                    | 148 ++++++++++++----------------------
 mm/userfaultfd.c              |  79 +++++++++---------
 4 files changed, 106 insertions(+), 156 deletions(-)
include/linux/shmem_fs.h +0 -14
index a8273b32e041..1a345142af7d 100644
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)
 
 extern bool shmem_charge(struct inode *inode, long pages);
 
-#ifdef CONFIG_USERFAULTFD
-#ifdef CONFIG_SHMEM
-extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
-				  struct vm_area_struct *dst_vma,
-				  unsigned long dst_addr,
-				  unsigned long src_addr,
-				  uffd_flags_t flags,
-				  struct folio **foliop);
-#else /* !CONFIG_SHMEM */
-#define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \
-			       src_addr, flags, foliop) ({ BUG(); 0; })
-#endif /* CONFIG_SHMEM */
-#endif /* CONFIG_USERFAULTFD */
-
 /*
  * Used space is stored as unsigned 64-bit value in bytes but
  * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +15 -6
index 4d8b879eed91..bf4e595ac914 100644
@@ -93,10 +93,24 @@ struct vm_uffd_ops {
 	struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
 	/*
 	 * Called during resolution of UFFDIO_COPY request.
-	 * Should return allocate a and return folio or NULL if allocation fails.
+	 * Should allocate and return a folio or NULL if allocation
+	 * fails.
 	 */
 	struct folio *(*alloc_folio)(struct vm_area_struct *vma,
 				     unsigned long addr);
+	/*
+	 * Called during resolution of UFFDIO_COPY request.
+	 * Should lock the folio and add it to VMA's page cache.
+	 * Returns 0 on success, error code on failure.
+	 */
+	int (*filemap_add)(struct folio *folio, struct vm_area_struct *vma,
+			 unsigned long addr);
+	/*
+	 * Called during resolution of UFFDIO_COPY request on the error
+	 * handling path.
+	 * Should revert the operation of ->filemap_add().
+	 */
+	void (*filemap_remove)(struct folio *folio, struct vm_area_struct *vma);
 };
 
 /* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
 /* Flags controlling behavior. These behavior changes are mode-independent. */
 #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
 
-extern int mfill_atomic_install_pte(pmd_t *dst_pmd,
-				    struct vm_area_struct *dst_vma,
-				    unsigned long dst_addr, struct page *page,
-				    bool newly_allocated, uffd_flags_t flags);
-
 extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 				 unsigned long src_start, unsigned long len,
 				 uffd_flags_t flags);
mm/shmem.c +53 -95

177 lines not on this page · raw message

mm/userfaultfd.c +38 -41

148 lines not on this page · raw message

Signature · 1 line
2.51.0
Reviewed-by on v2 10/15
#28 lore raw

Re: [PATCH v2 10/15] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

Quoted 56 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.

---
 include/linux/shmem_fs.h      |  14 ----
 include/linux/userfaultfd_k.h |  21 +++--
 mm/shmem.c                    | 148 ++++++++++++----------------------
 mm/userfaultfd.c              |  79 +++++++++---------
 4 files changed, 106 insertions(+), 156 deletions(-)
include/linux/shmem_fs.h +0 -14
index a8273b32e041..1a345142af7d 100644
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)
 
 extern bool shmem_charge(struct inode *inode, long pages);
 
-#ifdef CONFIG_USERFAULTFD
-#ifdef CONFIG_SHMEM
-extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
-                                 struct vm_area_struct *dst_vma,
-                                 unsigned long dst_addr,
-                                 unsigned long src_addr,
-                                 uffd_flags_t flags,
-                                 struct folio **foliop);
-#else /* !CONFIG_SHMEM */
-#define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \
-                              src_addr, flags, foliop) ({ BUG(); 0; })
-#endif /* CONFIG_SHMEM */
-#endif /* CONFIG_USERFAULTFD */
-
 /*
  * Used space is stored as unsigned 64-bit value in bytes but
  * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +0 -0
index 4d8b879eed91..bf4e595ac914 100644
@@ -93,10 +93,24 @@ struct vm_uffd_ops {
        struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
        /*

include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 103

Quoted 10 lines
include/linux/userfaultfd_k.h +5 -1
         * Called during resolution of UFFDIO_COPY request.
-        * Should return allocate a and return folio or NULL if allocation fails.
+        * Should allocate and return a folio or NULL if allocation
+        * fails.
         */
        struct folio *(*alloc_folio)(struct vm_area_struct *vma,
                                     unsigned long addr);
+       /*
+        * Called during resolution of UFFDIO_COPY request.
+        * Should lock the folio and add it to VMA's page cache.

I don't think "should lock the folio" is accurate. That sounds like "it will call folio_lock()" but it actually calls __folio_set_locked(). Maybe this is better:

"Should only be called with a folio returned by alloc_folio() above. The folio will set to locked."

Quoted 246 lines
include/linux/userfaultfd_k.h +10 -5
+        * Returns 0 on success, error code on failure.
+        */
+       int (*filemap_add)(struct folio *folio, struct vm_area_struct *vma,
+                        unsigned long addr);
+       /*
+        * Called during resolution of UFFDIO_COPY request on the error
+        * handling path.
+        * Should revert the operation of ->filemap_add().
+        */
+       void (*filemap_remove)(struct folio *folio, struct vm_area_struct *vma);
 };
 
 /* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
 /* Flags controlling behavior. These behavior changes are mode-independent. */
 #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
 
-extern int mfill_atomic_install_pte(pmd_t *dst_pmd,
-                                   struct vm_area_struct *dst_vma,
-                                   unsigned long dst_addr, struct page *page,
-                                   bool newly_allocated, uffd_flags_t flags);
-
 extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
                                 unsigned long src_start, unsigned long len,
                                 uffd_flags_t flags);
mm/shmem.c +53 -95
index 7bd887b64f62..68620caaf75f 100644
@@ -3181,118 +3181,73 @@ static struct inode *shmem_get_inode(struct mnt_idmap *idmap,
 #endif /* CONFIG_TMPFS_QUOTA */
 
 #ifdef CONFIG_USERFAULTFD
-int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
-                          struct vm_area_struct *dst_vma,
-                          unsigned long dst_addr,
-                          unsigned long src_addr,
-                          uffd_flags_t flags,
-                          struct folio **foliop)
-{
-       struct inode *inode = file_inode(dst_vma->vm_file);
-       struct shmem_inode_info *info = SHMEM_I(inode);
+static struct folio *shmem_mfill_folio_alloc(struct vm_area_struct *vma,
+                                            unsigned long addr)
+{
+       struct inode *inode = file_inode(vma->vm_file);
        struct address_space *mapping = inode->i_mapping;
+       struct shmem_inode_info *info = SHMEM_I(inode);
+       pgoff_t pgoff = linear_page_index(vma, addr);
        gfp_t gfp = mapping_gfp_mask(mapping);
-       pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
-       void *page_kaddr;
        struct folio *folio;
-       int ret;
-       pgoff_t max_off;
-
-       if (shmem_inode_acct_blocks(inode, 1)) {
-               /*
-                * We may have got a page, returned -ENOENT triggering a retry,
-                * and now we find ourselves with -ENOMEM. Release the page, to
-                * avoid a BUG_ON in our caller.
-                */
-               if (unlikely(*foliop)) {
-                       folio_put(*foliop);
-                       *foliop = NULL;
-               }
-               return -ENOMEM;
-       }
 
-       if (!*foliop) {
-               ret = -ENOMEM;
-               folio = shmem_alloc_folio(gfp, 0, info, pgoff);
-               if (!folio)
-                       goto out_unacct_blocks;
+       if (unlikely(pgoff >= DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE)))
+               return NULL;
 
-               if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY)) {
-                       page_kaddr = kmap_local_folio(folio, 0);
-                       /*
-                        * The read mmap_lock is held here.  Despite the
-                        * mmap_lock being read recursive a deadlock is still
-                        * possible if a writer has taken a lock.  For example:
-                        *
-                        * process A thread 1 takes read lock on own mmap_lock
-                        * process A thread 2 calls mmap, blocks taking write lock
-                        * process B thread 1 takes page fault, read lock on own mmap lock
-                        * process B thread 2 calls mmap, blocks taking write lock
-                        * process A thread 1 blocks taking read lock on process B
-                        * process B thread 1 blocks taking read lock on process A
-                        *
-                        * Disable page faults to prevent potential deadlock
-                        * and retry the copy outside the mmap_lock.
-                        */
-                       pagefault_disable();
-                       ret = copy_from_user(page_kaddr,
-                                            (const void __user *)src_addr,
-                                            PAGE_SIZE);
-                       pagefault_enable();
-                       kunmap_local(page_kaddr);
-
-                       /* fallback to copy_from_user outside mmap_lock */
-                       if (unlikely(ret)) {
-                               *foliop = folio;
-                               ret = -ENOENT;
-                               /* don't free the page */
-                               goto out_unacct_blocks;
-                       }
+       folio = shmem_alloc_folio(gfp, 0, info, pgoff);
+       if (!folio)
+               return NULL;
 
-                       flush_dcache_folio(folio);
-               } else {                /* ZEROPAGE */
-                       clear_user_highpage(&folio->page, dst_addr);
-               }
-       } else {
-               folio = *foliop;
-               VM_BUG_ON_FOLIO(folio_test_large(folio), folio);
-               *foliop = NULL;
+       if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) {
+               folio_put(folio);
+               return NULL;
        }
 
-       VM_BUG_ON(folio_test_locked(folio));
-       VM_BUG_ON(folio_test_swapbacked(folio));
+       return folio;
+}
+
+static int shmem_mfill_filemap_add(struct folio *folio,
+                                  struct vm_area_struct *vma,
+                                  unsigned long addr)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+       struct address_space *mapping = inode->i_mapping;
+       pgoff_t pgoff = linear_page_index(vma, addr);
+       gfp_t gfp = mapping_gfp_mask(mapping);
+       int err;
+
        __folio_set_locked(folio);
        __folio_set_swapbacked(folio);
-       __folio_mark_uptodate(folio);
-
-       ret = -EFAULT;
-       max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
-       if (unlikely(pgoff >= max_off))
-               goto out_release;
 
-       ret = mem_cgroup_charge(folio, dst_vma->vm_mm, gfp);
-       if (ret)
-               goto out_release;
-       ret = shmem_add_to_page_cache(folio, mapping, pgoff, NULL, gfp);
-       if (ret)
-               goto out_release;
+       err = shmem_add_to_page_cache(folio, mapping, pgoff, NULL, gfp);
+       if (err)
+               goto err_unlock;
 
-       ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
-                                      &folio->page, true, flags);
-       if (ret)
-               goto out_delete_from_cache;
+       if (shmem_inode_acct_blocks(inode, 1)) {
+               err = -ENOMEM;
+               goto err_delete_from_cache;
+       }
 
+       folio_add_lru(folio);
        shmem_recalc_inode(inode, 1, 0);
-       folio_unlock(folio);
+
        return 0;
-out_delete_from_cache:
+
+err_delete_from_cache:
        filemap_remove_folio(folio);
-out_release:
+err_unlock:
+       folio_unlock(folio);
+       return err;
+}
+
+static void shmem_mfill_filemap_remove(struct folio *folio,
+                                      struct vm_area_struct *vma)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+
+       filemap_remove_folio(folio);
+       shmem_recalc_inode(inode, 0, 0);
        folio_unlock(folio);
-       folio_put(folio);
-out_unacct_blocks:
-       shmem_inode_unacct_blocks(inode, 1);
-       return ret;
 }
 
 static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
@@ -3315,6 +3270,9 @@ static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
 static const struct vm_uffd_ops shmem_uffd_ops = {
        .can_userfault          = shmem_can_userfault,
        .get_folio_noalloc      = shmem_get_folio_noalloc,
+       .alloc_folio            = shmem_mfill_folio_alloc,
+       .filemap_add            = shmem_mfill_filemap_add,
+       .filemap_remove         = shmem_mfill_filemap_remove,
 };
 #endif /* CONFIG_USERFAULTFD */
 
mm/userfaultfd.c +4 -8
index 31f3ab6a73e2..a0f8e67006d6 100644
@@ -14,7 +14,6 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/mmu_notifier.h>
 #include <linux/hugetlb.h>
-#include <linux/shmem_fs.h>
 #include <asm/tlbflush.h>
 #include <asm/tlb.h>
 #include "internal.h"
@@ -340,10 +339,10 @@ static bool mfill_file_over_size(struct vm_area_struct *dst_vma,
  * This function handles both MCOPY_ATOMIC_NORMAL and _CONTINUE for both shmem
  * and anon, and for both shared and private VMAs.
  */
-int mfill_atomic_install_pte(pmd_t *dst_pmd,
-                            struct vm_area_struct *dst_vma,
-                            unsigned long dst_addr, struct page *page,
-                            bool newly_allocated, uffd_flags_t flags)
+static int mfill_atomic_install_pte(pmd_t *dst_pmd,
+                                   struct vm_area_struct *dst_vma,
+                                   unsigned long dst_addr, struct page *page,
+                                   uffd_flags_t flags)
 {
        int ret;
        struct mm_struct *dst_mm = dst_vma->vm_mm;
@@ -387,9 +386,6 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,
                goto out_unlock;
 
        if (page_in_cache) {
-               /* Usually, cache pages are already added to LRU */
-               if (newly_allocated)
-                       folio_add_lru(folio);
                folio_add_file_rmap_pte(folio, page, dst_vma);
        } else {
                folio_add_new_anon_rmap(folio, dst_vma, dst_addr, RMAP_EXCLUSIVE);

mm/userfaultfd.c int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 404

Quoted 5 lines
mm/userfaultfd.c +2 -0
@@ -404,6 +400,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,
 
        set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte);
 
+       if (page_in_cache)
+               folio_unlock(folio);

I don't really like doing the folio_unlock() *here*, I think it's clearer if the callers (mfill_atomic_pte_continue() and __mfill_atomic_pte()) unlocked the folio themselves. But that's just my opinion.

Quoted 81 lines
mm/userfaultfd.c +27 -10
+
        /* No need to invalidate - it was non-present before */
        update_mmu_cache(dst_vma, dst_addr, dst_pte);
        ret = 0;
@@ -516,13 +515,22 @@ static int __mfill_atomic_pte(struct mfill_state *state,
         */
        __folio_mark_uptodate(folio);
 
+       if (ops->filemap_add) {
+               ret = ops->filemap_add(folio, state->vma, state->dst_addr);
+               if (ret)
+                       goto err_folio_put;
+       }
+
        ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr,
-                                      &folio->page, true, flags);
+                                      &folio->page, flags);
        if (ret)
-               goto err_folio_put;
+               goto err_filemap_remove;
 
        return 0;
 
+err_filemap_remove:
+       if (ops->filemap_remove)
+               ops->filemap_remove(folio, state->vma);
 err_folio_put:
        folio_put(folio);
        /* Don't return -ENOENT so that our caller won't retry */
@@ -535,6 +543,18 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
        const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
 
+       /*
+        * The normal page fault path for a MAP_PRIVATE mapping in a
+        * file-backed VMA will invoke the fault, fill the hole in the file and
+        * COW it right away. The result generates plain anonymous memory.
+        * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll
+        * generate anonymous memory directly without actually filling the
+        * hole. For the MAP_PRIVATE case the robustness check only happens in
+        * the pagetable (to verify it's still none) and not in the page cache.
+        */
+       if (!(state->vma->vm_flags & VM_SHARED))
+               ops = &anon_uffd_ops;
+
        return __mfill_atomic_pte(state, ops);
 }
 
@@ -554,7 +574,8 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
        spinlock_t *ptl;
        int ret;
 
-       if (mm_forbids_zeropage(dst_vma->vm_mm))
+       if (mm_forbids_zeropage(dst_vma->vm_mm) ||
+           (dst_vma->vm_flags & VM_SHARED))
                return mfill_atomic_pte_zeroed_folio(state);
 
        _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr),
@@ -609,11 +630,10 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
        }
 
        ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
-                                      page, false, flags);
+                                      page, flags);
        if (ret)
                goto out_release;
 
-       folio_unlock(folio);
        return 0;
 
 out_release:
@@ -836,41 +856,18 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx,
 
 static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 {
-       struct vm_area_struct *dst_vma = state->vma;
-       unsigned long src_addr = state->src_addr;
-       unsigned long dst_addr = state->dst_addr;
-       struct folio **foliop = &state->folio;
        uffd_flags_t flags = state->flags;
-       pmd_t *dst_pmd = state->pmd;

mm/userfaultfd.c extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 868

Quoted 10 lines
mm/userfaultfd.c +4 -1
-       ssize_t err;
 
        if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
                return mfill_atomic_pte_continue(state);
        if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
                return mfill_atomic_pte_poison(state);
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
+               return mfill_atomic_pte_copy(state);
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE))
+               return mfill_atomic_pte_zeropage(state);

Thanks for this cleanup. :)

Quoted 14 lines
mm/userfaultfd.c +0 -13
-       /*
-        * The normal page fault path for a shmem will invoke the
-        * fault, fill the hole in the file and COW it right away. The
-        * result generates plain anonymous memory. So when we are
-        * asked to fill an hole in a MAP_PRIVATE shmem mapping, we'll
-        * generate anonymous memory directly without actually filling
-        * the hole. For the MAP_PRIVATE case the robustness check
-        * only happens in the pagetable (to verify it's still none)
-        * and not in the radix tree.
-        */
-       if (!(dst_vma->vm_flags & VM_SHARED)) {
-               if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-                       err = mfill_atomic_pte_copy(state);

mm/userfaultfd.c extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 870

Quoted 10 lines
mm/userfaultfd.c +1 -9
-               else
-                       err = mfill_atomic_pte_zeropage(state);
-       } else {
-               err = shmem_mfill_atomic_pte(dst_pmd, dst_vma,
-                                            dst_addr, src_addr,
-                                            flags, foliop);
-       }
-
-       return err;
+       return -EOPNOTSUPP;

WARN_ONCE() here I think.

Feel free to add:

author maintainer
#29 lore raw

Re: [PATCH v2 10/15] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

Quoted 72 lines · James Houghton
Quoted 56 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.

---
 include/linux/shmem_fs.h      |  14 ----
 include/linux/userfaultfd_k.h |  21 +++--
 mm/shmem.c                    | 148 ++++++++++++----------------------
 mm/userfaultfd.c              |  79 +++++++++---------
 4 files changed, 106 insertions(+), 156 deletions(-)
include/linux/shmem_fs.h +0 -14
index a8273b32e041..1a345142af7d 100644
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)
 
 extern bool shmem_charge(struct inode *inode, long pages);
 
-#ifdef CONFIG_USERFAULTFD
-#ifdef CONFIG_SHMEM
-extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
-                                 struct vm_area_struct *dst_vma,
-                                 unsigned long dst_addr,
-                                 unsigned long src_addr,
-                                 uffd_flags_t flags,
-                                 struct folio **foliop);
-#else /* !CONFIG_SHMEM */
-#define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \
-                              src_addr, flags, foliop) ({ BUG(); 0; })
-#endif /* CONFIG_SHMEM */
-#endif /* CONFIG_USERFAULTFD */
-
 /*
  * Used space is stored as unsigned 64-bit value in bytes but
  * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +0 -0
index 4d8b879eed91..bf4e595ac914 100644
@@ -93,10 +93,24 @@ struct vm_uffd_ops {
        struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
        /*

include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 103

Quoted 10 lines
include/linux/userfaultfd_k.h +5 -1
         * Called during resolution of UFFDIO_COPY request.
-        * Should return allocate a and return folio or NULL if allocation fails.
+        * Should allocate and return a folio or NULL if allocation
+        * fails.
         */
        struct folio *(*alloc_folio)(struct vm_area_struct *vma,
                                     unsigned long addr);
+       /*
+        * Called during resolution of UFFDIO_COPY request.
+        * Should lock the folio and add it to VMA's page cache.

I don't think "should lock the folio" is accurate. That sounds like "it will call folio_lock()" but it actually calls __folio_set_locked(). Maybe this is better:

Quoted 2 lines

"Should only be called with a folio returned by alloc_folio() above. The folio will set to locked."

Yeah, sounds good.

Quoted 9 lines
Quoted 9 lines
include/linux/userfaultfd_k.h +4 -0
+        * Returns 0 on success, error code on failure.
+        */
+       int (*filemap_add)(struct folio *folio, struct vm_area_struct *vma,
+                        unsigned long addr);
 
@@ -404,6 +400,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,
 
        set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte);
 
Quoted 7 lines

include/linux/userfaultfd_k.h int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 404

Quoted 2 lines
include/linux/userfaultfd_k.h +2 -0
+       if (page_in_cache)
+               folio_unlock(folio);

I don't really like doing the folio_unlock() *here*, I think it's clearer if the callers (mfill_atomic_pte_continue() and __mfill_atomic_pte()) unlocked the folio themselves. But that's just my opinion.

We already have page_in_cache here, so I'd prefer to keep a single folio_unlock() rather than add additional if (page_in_cache) in __mfill_atomic_pte().

Quoted 52 lines
Quoted 14 lines
include/linux/userfaultfd_k.h +1 -5
+
        /* No need to invalidate - it was non-present before */
        update_mmu_cache(dst_vma, dst_addr, dst_pte);
        ret = 0;
@@ -836,41 +856,18 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx,
 
 static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 {
-       struct vm_area_struct *dst_vma = state->vma;
-       unsigned long src_addr = state->src_addr;
-       unsigned long dst_addr = state->dst_addr;
-       struct folio **foliop = &state->folio;
        uffd_flags_t flags = state->flags;
-       pmd_t *dst_pmd = state->pmd;

include/linux/userfaultfd_k.h extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 868

Quoted 10 lines
include/linux/userfaultfd_k.h +4 -1
-       ssize_t err;
 
        if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
                return mfill_atomic_pte_continue(state);
        if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
                return mfill_atomic_pte_poison(state);
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
+               return mfill_atomic_pte_copy(state);
+       if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE))
+               return mfill_atomic_pte_zeropage(state);

Thanks for this cleanup. :)

Quoted 14 lines
include/linux/userfaultfd_k.h +0 -13
-       /*
-        * The normal page fault path for a shmem will invoke the
-        * fault, fill the hole in the file and COW it right away. The
-        * result generates plain anonymous memory. So when we are
-        * asked to fill an hole in a MAP_PRIVATE shmem mapping, we'll
-        * generate anonymous memory directly without actually filling
-        * the hole. For the MAP_PRIVATE case the robustness check
-        * only happens in the pagetable (to verify it's still none)
-        * and not in the radix tree.
-        */
-       if (!(dst_vma->vm_flags & VM_SHARED)) {
-               if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-                       err = mfill_atomic_pte_copy(state);

include/linux/userfaultfd_k.h extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 869

Quoted 10 lines
include/linux/userfaultfd_k.h +1 -9
-               else
-                       err = mfill_atomic_pte_zeropage(state);
-       } else {
-               err = shmem_mfill_atomic_pte(dst_pmd, dst_vma,
-                                            dst_addr, src_addr,
-                                            flags, foliop);
-       }
-
-       return err;
+       return -EOPNOTSUPP;
Quoted 1 line

WARN_ONCE() here I think.

I'll add VM_WARN_ONCE() here.

Quoted 3 lines

Feel free to add:

Thanks!

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#30 lore raw

PATCH v2 11/15 userfaultfd: mfill_atomic(): remove retry logic

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().

Drop the retry logic from mfill_atomic().

---
 mm/userfaultfd.c | 24 ------------------------
 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
index a0f8e67006d6..7cd7c5d1ce84 100644
@@ -29,7 +29,6 @@ struct mfill_state {
 	struct vm_area_struct *vma;
 	unsigned long src_addr;
 	unsigned long dst_addr;
-	struct folio *folio;
 	pmd_t *pmd;
 };
 
@@ -898,7 +897,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(src_start + len <= src_start);
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
-retry:
 	err = mfill_get_vma(&state);
 	if (err)
 		goto out;
@@ -925,26 +923,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		err = mfill_atomic_pte(&state);
 		cond_resched();
 
-		if (unlikely(err == -ENOENT)) {
-			void *kaddr;
-
-			mfill_put_vma(&state);
-			VM_WARN_ON_ONCE(!state.folio);
-
-			kaddr = kmap_local_folio(state.folio, 0);
-			err = copy_from_user(kaddr,
-					     (const void __user *)state.src_addr,
-					     PAGE_SIZE);
-			kunmap_local(kaddr);
-			if (unlikely(err)) {
-				err = -EFAULT;
-				goto out;
-			}
-			flush_dcache_folio(state.folio);
-			goto retry;
-		} else
-			VM_WARN_ON_ONCE(state.folio);
-
 		if (!err) {
 			state.dst_addr += PAGE_SIZE;
 			state.src_addr += PAGE_SIZE;
@@ -959,8 +937,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
 	mfill_put_vma(&state);
 out:
-	if (state.folio)
-		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);
Signature · 1 line
2.51.0
author maintainer
#31 lore raw

PATCH v2 12/15 mm: generalize handling of userfaults in __do_fault()

In reply to Mike Rapoport

From: Peter Xu <[email protected]>

When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:

- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist

Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().

For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.

Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.

---
 mm/memory.c      | 43 +++++++++++++++++++++++++++++++++++++++++++
 mm/shmem.c       | 12 ------------
 mm/userfaultfd.c |  8 ++++++++
 3 files changed, 51 insertions(+), 12 deletions(-)
mm/memory.c +43 -0
index 07778814b4a8..e2183c44d70b 100644
@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	return VM_FAULT_OOM;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct inode *inode;
+	struct folio *folio;
+
+	if (!(userfaultfd_missing(vma) || userfaultfd_minor(vma)))
+		return 0;
+
+	inode = file_inode(vma->vm_file);
+	folio = vma->vm_ops->uffd_ops->get_folio_noalloc(inode, vmf->pgoff);
+	if (!IS_ERR_OR_NULL(folio)) {
+		/*
+		 * TODO: provide a flag for get_folio_noalloc() to avoid
+		 * locking (or even the extra reference?)
+		 */
+		folio_unlock(folio);
+		folio_put(folio);
+		if (userfaultfd_minor(vma))
+			return handle_userfault(vmf, VM_UFFD_MINOR);
+	} else {
+		if (userfaultfd_missing(vma))
+			return handle_userfault(vmf, VM_UFFD_MISSING);
+	}
+
+	return 0;
+}
+#else
+static inline vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+	return 0;
+}
+#endif
+
 /*
  * The mmap_lock must have been held on entry, and may have been
  * released depending on flags and vma->vm_ops->fault() return value.
@@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
 			return VM_FAULT_OOM;
 	}
 
+	/*
+	 * If this is an userfaultfd trap, process it in advance before
+	 * triggering the genuine fault handler.
+	 */
+	ret = __do_userfault(vmf);
+	if (ret)
+		return ret;
+
 	ret = vma->vm_ops->fault(vmf);
 	if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
 			    VM_FAULT_DONE_COW)))
mm/shmem.c +0 -12
index 68620caaf75f..239545352cd2 100644
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	fault_mm = vma ? vma->vm_mm : NULL;
 
 	folio = filemap_get_entry(inode->i_mapping, index);
-	if (folio && vma && userfaultfd_minor(vma)) {
-		if (!xa_is_value(folio))
-			folio_put(folio);
-		*fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
-		return 0;
-	}
-
 	if (xa_is_value(folio)) {
 		error = shmem_swapin_folio(inode, index, &folio,
 					   sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	 * Fast cache lookup and swap lookup did not find it: allocate.
 	 */
 
-	if (vma && userfaultfd_missing(vma)) {
-		*fault_type = handle_userfault(vmf, VM_UFFD_MISSING);
-		return 0;
-	}
-
 	/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */
 	orders = shmem_allowable_huge_orders(inode, vma, index, write_end, false);
 	if (orders > 0) {
mm/userfaultfd.c +8 -0
index 7cd7c5d1ce84..2ac5fad0ed6c 100644
@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 	    !vma_is_anonymous(vma))
 		return false;
 
+	/*
+	 * File backed memory with PTE level mappigns must implement
+	 * ops->get_folio_noalloc()
+	 */
+	if (!vma_is_anonymous(vma) && !is_vm_hugetlb_page(vma) &&
+	    !ops->get_folio_noalloc)
+		return false;
+
 	return ops->can_userfault(vma, vm_flags);
 }
 
Signature · 1 line
2.51.0
#32 lore raw

Re: [PATCH v2 12/15] mm: generalize handling of userfaults in __do_fault()

Quoted 41 lines · Mike Rapoport

From: Peter Xu <[email protected]>

When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:

- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist

Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().

For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.

Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.

---
 mm/memory.c      | 43 +++++++++++++++++++++++++++++++++++++++++++
 mm/shmem.c       | 12 ------------
 mm/userfaultfd.c |  8 ++++++++
 3 files changed, 51 insertions(+), 12 deletions(-)
mm/memory.c +2 -0
index 07778814b4a8..e2183c44d70b 100644
@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        return VM_FAULT_OOM;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static vm_fault_t __do_userfault(struct vm_fault *vmf)

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5342

Quoted 10 lines
mm/memory.c +10 -0
+{
+       struct vm_area_struct *vma = vmf->vma;
+       struct inode *inode;
+       struct folio *folio;
+
+       if (!(userfaultfd_missing(vma) || userfaultfd_minor(vma)))
+               return 0;
+
+       inode = file_inode(vma->vm_file);
+       folio = vma->vm_ops->uffd_ops->get_folio_noalloc(inode, vmf->pgoff);

If you do away with the change you made to vma_can_userfault(), please add a WARN_ON_ONCE() here if uffd_ops or uffd_ops->get_folio_noalloc are not present.

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5347

Quoted 5 lines
mm/memory.c +5 -0
+       if (!IS_ERR_OR_NULL(folio)) {
+               /*
+                * TODO: provide a flag for get_folio_noalloc() to avoid
+                * locking (or even the extra reference?)
+                */

I think a whole new op is better than adding a flag. Something like uffd_ops->folio_present().

Quoted 22 lines
mm/memory.c +18 -0
+               folio_unlock(folio);
+               folio_put(folio);
+               if (userfaultfd_minor(vma))
+                       return handle_userfault(vmf, VM_UFFD_MINOR);
+       } else {
+               if (userfaultfd_missing(vma))
+                       return handle_userfault(vmf, VM_UFFD_MISSING);
+       }
+
+       return 0;
+}
+#else
+static inline vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+       return 0;
+}
+#endif
+
 /*
  * The mmap_lock must have been held on entry, and may have been
  * released depending on flags and vma->vm_ops->fault() return value.

mm/memory.c static vm_fault_t __do_fault(struct vm_fault *vmf) · line 5399

Quoted 5 lines
mm/memory.c +2 -0
@@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
                        return VM_FAULT_OOM;
        }
 
+       /*
+        * If this is an userfaultfd trap, process it in advance before

"If this is a userfault"

Quoted 44 lines
mm/memory.c +6 -0
+        * triggering the genuine fault handler.
+        */
+       ret = __do_userfault(vmf);
+       if (ret)
+               return ret;
+
        ret = vma->vm_ops->fault(vmf);
        if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
                            VM_FAULT_DONE_COW)))
mm/shmem.c +0 -12
index 68620caaf75f..239545352cd2 100644
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
        fault_mm = vma ? vma->vm_mm : NULL;
 
        folio = filemap_get_entry(inode->i_mapping, index);
-       if (folio && vma && userfaultfd_minor(vma)) {
-               if (!xa_is_value(folio))
-                       folio_put(folio);
-               *fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
-               return 0;
-       }
-
        if (xa_is_value(folio)) {
                error = shmem_swapin_folio(inode, index, &folio,
                                           sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
         * Fast cache lookup and swap lookup did not find it: allocate.
         */
 
-       if (vma && userfaultfd_missing(vma)) {
-               *fault_type = handle_userfault(vmf, VM_UFFD_MISSING);
-               return 0;
-       }
-
        /* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */
        orders = shmem_allowable_huge_orders(inode, vma, index, write_end, false);
        if (orders > 0) {
mm/userfaultfd.c +0 -0
index 7cd7c5d1ce84..2ac5fad0ed6c 100644

mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2049

Quoted 5 lines
mm/userfaultfd.c +2 -0
@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
            !vma_is_anonymous(vma))
                return false;
 
+       /*
+        * File backed memory with PTE level mappigns must implement

"File-backed VMAs (except HugeTLB VMAs) must implement ops->get_folio_alloc() to support userfaults, as all userfaults in file-backed VMAs wlll call ops->get_folio_alloc() to determine the userfault type."

mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2053

Quoted 4 lines
mm/userfaultfd.c +4 -0
+        * ops->get_folio_noalloc()
+        */
+       if (!vma_is_anonymous(vma) && !is_vm_hugetlb_page(vma) &&
+           !ops->get_folio_noalloc)

With the separate folio_present() (or whatever) op, this check becomes more obvious. (Looking at this right now, it looks like we're checking for UFFDIO_CONTINUE/minor fault support here, but that's not really what's going on.)

Honestly I'm not 100% sure if this check should be here. If it's going to be here, then we should probably also have checks like these here:

if (vm_flags & VM_UFFD_MINOR && !ops->get_folio_noalloc)

and

if (vm_flags & VM_UFFD_MISSING && !ops->alloc_folio)

So, maybe we should just leave it up to ops->can_userfault() to make sure we're doing the right thing?

Quoted 8 lines
mm/userfaultfd.c +2 -1
+               return false;
+
        return ops->can_userfault(vma, vm_flags);
 }
 
--

2.51.0

author maintainer
#33 lore raw

Re: [PATCH v2 12/15] mm: generalize handling of userfaults in __do_fault()

Quoted 57 lines · James Houghton
Quoted 41 lines · Mike Rapoport

From: Peter Xu <[email protected]>

When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:

- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist

Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().

For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.

Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.

---
 mm/memory.c      | 43 +++++++++++++++++++++++++++++++++++++++++++
 mm/shmem.c       | 12 ------------
 mm/userfaultfd.c |  8 ++++++++
 3 files changed, 51 insertions(+), 12 deletions(-)
mm/memory.c +2 -0
index 07778814b4a8..e2183c44d70b 100644
@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        return VM_FAULT_OOM;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static vm_fault_t __do_userfault(struct vm_fault *vmf)

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5342

Quoted 10 lines
mm/memory.c +10 -0
+{
+       struct vm_area_struct *vma = vmf->vma;
+       struct inode *inode;
+       struct folio *folio;
+
+       if (!(userfaultfd_missing(vma) || userfaultfd_minor(vma)))
+               return 0;
+
+       inode = file_inode(vma->vm_file);
+       folio = vma->vm_ops->uffd_ops->get_folio_noalloc(inode, vmf->pgoff);

If you do away with the change you made to vma_can_userfault(), please add a WARN_ON_ONCE() here if uffd_ops or uffd_ops->get_folio_noalloc are not present.

Quoted 8 lines

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5347

Quoted 5 lines
mm/memory.c +5 -0
+       if (!IS_ERR_OR_NULL(folio)) {
+               /*
+                * TODO: provide a flag for get_folio_noalloc() to avoid
+                * locking (or even the extra reference?)
+                */

I think a whole new op is better than adding a flag. Something like uffd_ops->folio_present().

Right now it follows what shmem does, let's keep it as TODO for future optimizations.

Quoted 25 lines
Quoted 25 lines
mm/memory.c +18 -0
+               folio_unlock(folio);
+               folio_put(folio);
+               if (userfaultfd_minor(vma))
+                       return handle_userfault(vmf, VM_UFFD_MINOR);
+       } else {
+               if (userfaultfd_missing(vma))
+                       return handle_userfault(vmf, VM_UFFD_MISSING);
+       }
+
+       return 0;
+}
+#else
+static inline vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+       return 0;
+}
+#endif
+
 /*
  * The mmap_lock must have been held on entry, and may have been
  * released depending on flags and vma->vm_ops->fault() return value.
@@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
                        return VM_FAULT_OOM;
        }
 
Quoted 4 lines

mm/memory.c static vm_fault_t __do_fault(struct vm_fault *vmf) · line 5399

Quoted 2 lines
mm/memory.c +2 -0
+       /*
+        * If this is an userfaultfd trap, process it in advance before

"If this is a userfault"

Indeed :)

Quoted 47 lines
Quoted 47 lines
mm/memory.c +6 -0
+        * triggering the genuine fault handler.
+        */
+       ret = __do_userfault(vmf);
+       if (ret)
+               return ret;
+
        ret = vma->vm_ops->fault(vmf);
        if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
                            VM_FAULT_DONE_COW)))
mm/shmem.c +0 -12
index 68620caaf75f..239545352cd2 100644
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
        fault_mm = vma ? vma->vm_mm : NULL;
 
        folio = filemap_get_entry(inode->i_mapping, index);
-       if (folio && vma && userfaultfd_minor(vma)) {
-               if (!xa_is_value(folio))
-                       folio_put(folio);
-               *fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
-               return 0;
-       }
-
        if (xa_is_value(folio)) {
                error = shmem_swapin_folio(inode, index, &folio,
                                           sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
         * Fast cache lookup and swap lookup did not find it: allocate.
         */
 
-       if (vma && userfaultfd_missing(vma)) {
-               *fault_type = handle_userfault(vmf, VM_UFFD_MISSING);
-               return 0;
-       }
-
        /* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */
        orders = shmem_allowable_huge_orders(inode, vma, index, write_end, false);
        if (orders > 0) {
mm/userfaultfd.c +0 -0
index 7cd7c5d1ce84..2ac5fad0ed6c 100644
@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
            !vma_is_anonymous(vma))
                return false;
 
Quoted 7 lines

mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2049

Quoted 2 lines
mm/userfaultfd.c +2 -0
+       /*
+        * File backed memory with PTE level mappigns must implement

"File-backed VMAs (except HugeTLB VMAs) must implement ops->get_folio_alloc() to support userfaults, as all userfaults in file-backed VMAs wlll call ops->get_folio_alloc() to determine the userfault type."

I like (except HugeTLB) more than "PTE ...", but it's not here to determine the userfaulfd type, it's here to ensure we can __do_userfault() for a memory type.

/*
* File backed VMAs (except HugeTLB) must implement
* ops->get_folio_noalloc() because it's required by
* __do_userfault() in page fault handling. */

Quoted 13 lines

mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2053

Quoted 4 lines
mm/userfaultfd.c +4 -0
+        * ops->get_folio_noalloc()
+        */
+       if (!vma_is_anonymous(vma) && !is_vm_hugetlb_page(vma) &&
+           !ops->get_folio_noalloc)

With the separate folio_present() (or whatever) op, this check becomes more obvious. (Looking at this right now, it looks like we're checking for UFFDIO_CONTINUE/minor fault support here, but that's not really what's going on.)

Honestly I'm not 100% sure if this check should be here. If it's going to be here, then we should probably also have checks like these here:

Quoted 8 lines

if (vm_flags & VM_UFFD_MINOR && !ops->get_folio_noalloc)

and

if (vm_flags & VM_UFFD_MISSING && !ops->alloc_folio)

So, maybe we should just leave it up to ops->can_userfault() to make sure we're doing the right thing?

This must be in the core so it can reject VMAs that can't be used in __do_userfault() without complicating hot path in the page fault handler with extra conditions.

Quoted 9 lines
Quoted 9 lines
mm/userfaultfd.c +2 -1
+               return false;
+
        return ops->can_userfault(vma, vm_flags);
 }
 
--

2.51.0

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#34 lore raw

PATCH v2 13/15 KVM: guest_memfd: implement userfaultfd operations

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

---
 mm/filemap.c           |  1 +
 virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
index 6cd7974d4ada..19dfcebcd23f 100644
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)
 
 	filemap_free_folio(mapping, folio);
 }
+EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
 
 /*
  * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +82 -2

114 lines not on this page · raw message

Signature · 1 line
2.51.0
#35 lore raw

Re: [PATCH v2 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 11 lines · Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").

Quoted 8 lines

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

Overall looks fine to me, but I am slightly concerned about in-place conversion[1], and I think you're going to want to implement a kvm_gmem_folio_present() op or something (like I was saying on the previous patch[2]).

[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVUJ5FL97d9ytxp2WXos6HS+U+ycpsi5VxffsW9vacr9Q@mail.gmail.com/

Some in-line comments below.

Quoted 30 lines
---
 mm/filemap.c           |  1 +
 virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
index 6cd7974d4ada..19dfcebcd23f 100644
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)
 
        filemap_free_folio(mapping, folio);
 }
+EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
 
 /*
  * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +1 -0
index 017d84a7adf3..46582feeed75 100644
@@ -7,6 +7,7 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include <linux/userfaultfd_k.h>
 
 #include "kvm_mm.h"
 

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 115

Quoted 8 lines
virt/kvm/guest_memfd.c +5 -0
@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
        return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
 }
 
+static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+       return __filemap_get_folio(inode->i_mapping, pgoff,
+                                  FGP_LOCK | FGP_ACCESSED, 0);
+}

When in-place conversion is supported, I wonder what the semantics should be for when we get userfaults.

Upon a userspace access to a file offset that is populated but private, should we get a userfault or a SIGBUS?

I guess getting a userfault is strictly more useful for userspace, but I'm not sure which choice is more correct.

Quoted 55 lines
virt/kvm/guest_memfd.c +39 -2
+
 /*
  * Returns a locked folio on success.  The caller is responsible for
  * setting the up-to-date flag before the memory is mapped into the guest.
@@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
         * Fast-path: See if folio is already present in mapping to avoid
         * policy_lookup.
         */
-       folio = __filemap_get_folio(inode->i_mapping, index,
-                                   FGP_LOCK | FGP_ACCESSED, 0);
+       folio = kvm_gmem_get_folio_noalloc(inode, index);
        if (!IS_ERR(folio))
                return folio;
 
@@ -457,12 +463,86 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
 }
 #endif /* CONFIG_NUMA */
 
+#ifdef CONFIG_USERFAULTFD
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+
+       /*
+        * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+        * This ensures the memory can be mapped to userspace.
+        */
+       if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+               return false;
+
+       return true;
+}
+
+static struct folio *kvm_gmem_folio_alloc(struct vm_area_struct *vma,
+                                         unsigned long addr)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+       pgoff_t pgoff = linear_page_index(vma, addr);
+       struct mempolicy *mpol;
+       struct folio *folio;
+       gfp_t gfp;
+
+       if (unlikely(pgoff >= (i_size_read(inode) >> PAGE_SHIFT)))
+               return NULL;
+
+       gfp = mapping_gfp_mask(inode->i_mapping);
+       mpol = mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff);
+       mpol = mpol ?: get_task_policy(current);
+       folio = filemap_alloc_folio(gfp, 0, mpol);
+       mpol_cond_put(mpol);
+
+       return folio;
+}
+
+static int kvm_gmem_filemap_add(struct folio *folio,

virt/kvm/guest_memfd.c static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, · line 512

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+                               struct vm_area_struct *vma,
+                               unsigned long addr)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+       struct address_space *mapping = inode->i_mapping;
+       pgoff_t pgoff = linear_page_index(vma, addr);
+       int err;
+
+       __folio_set_locked(folio);
+       err = filemap_add_folio(mapping, folio, pgoff, GFP_KERNEL);

This is going to get more interesting with in-place conversion. I'm not really sure how to synchronize with it, but we'll probably need to take the invalidate lock for reading. And then we'll need a separate uffd_op to drop it after we install the PTE... I think.

Quoted 39 lines
virt/kvm/guest_memfd.c +27 -1
+       if (err) {
+               folio_unlock(folio);
+               return err;
+       }
+
+       return 0;
+}
+
+static void kvm_gmem_filemap_remove(struct folio *folio,
+                                   struct vm_area_struct *vma)
+{
+       filemap_remove_folio(folio);
+       folio_unlock(folio);
+}
+
+static const struct vm_uffd_ops kvm_gmem_uffd_ops = {
+       .can_userfault     = kvm_gmem_can_userfault,
+       .get_folio_noalloc = kvm_gmem_get_folio_noalloc,
+       .alloc_folio       = kvm_gmem_folio_alloc,
+       .filemap_add       = kvm_gmem_filemap_add,
+       .filemap_remove    = kvm_gmem_filemap_remove,
+};
+#endif /* CONFIG_USERFAULTFD */
+
 static const struct vm_operations_struct kvm_gmem_vm_ops = {
        .fault          = kvm_gmem_fault_user_mapping,
 #ifdef CONFIG_NUMA
        .get_policy     = kvm_gmem_get_policy,
        .set_policy     = kvm_gmem_set_policy,
 #endif
+#ifdef CONFIG_USERFAULTFD
+       .uffd_ops       = &kvm_gmem_uffd_ops,
+#endif
 };
 
 static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
--

2.51.0

author maintainer
#36 lore raw

Re: [PATCH v2 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 17 lines · James Houghton
Quoted 16 lines · Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

Quoted 8 lines
Quoted 3 lines

Overall looks fine to me, but I am slightly concerned about in-place conversion[1], and I think you're going to want to implement a kvm_gmem_folio_present() op or something (like I was saying on the previous patch[2]).

Let's solve each problem in it's time :)

Quoted 27 lines

[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVUJ5FL97d9ytxp2WXos6HS+U+ycpsi5VxffsW9vacr9Q@mail.gmail.com/

Some in-line comments below.

Quoted 10 lines
---
 mm/filemap.c           |  1 +
 virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +0 -0
index 6cd7974d4ada..19dfcebcd23f 100644

mm/filemap.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 115

Quoted 8 lines
mm/filemap.c +5 -0
@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
        return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
 }
 
+static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+       return __filemap_get_folio(inode->i_mapping, pgoff,
+                                  FGP_LOCK | FGP_ACCESSED, 0);
+}

When in-place conversion is supported, I wonder what the semantics should be for when we get userfaults.

Quoted 5 lines

Upon a userspace access to a file offset that is populated but private, should we get a userfault or a SIGBUS?

I guess getting a userfault is strictly more useful for userspace, but I'm not sure which choice is more correct.

Me neither :)

We can deliver userfault, but just block UFFDIO_COPY, can't we?

Quoted 8 lines
Quoted 8 lines
mm/filemap.c +8 -0
+static int kvm_gmem_filemap_add(struct folio *folio,
+                               struct vm_area_struct *vma,
+                               unsigned long addr)
+{
+       struct inode *inode = file_inode(vma->vm_file);
+       struct address_space *mapping = inode->i_mapping;
+       pgoff_t pgoff = linear_page_index(vma, addr);
+       int err;
Quoted 8 lines

mm/filemap.c

Quoted 3 lines
mm/filemap.c +3 -0
+
+       __folio_set_locked(folio);
+       err = filemap_add_folio(mapping, folio, pgoff, GFP_KERNEL);

This is going to get more interesting with in-place conversion. I'm not really sure how to synchronize with it, but we'll probably need to take the invalidate lock for reading. And then we'll need a separate uffd_op to drop it after we install the PTE... I think.

I think we can start simple and then move on along with the in-place conversion work. If there will be a need for a new uffd_ops callback we can add it then.

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#37 lore raw

PATCH v2 14/15 KVM: selftests: test userfaultfd minor for guest_memfd

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.

---
 .../testing/selftests/kvm/guest_memfd_test.c  | 113 ++++++++++++++++++
 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
index 618c937f3c90..7612819e340a 100644
@@ -10,13 +10,17 @@
 #include <errno.h>
 #include <stdio.h>
 #include <fcntl.h>
+#include <pthread.h>
 
 #include <linux/bitmap.h>
 #include <linux/falloc.h>
 #include <linux/sizes.h>
+#include <linux/userfaultfd.h>
 #include <sys/mman.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/ioctl.h>
 
 #include "kvm_util.h"
 #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)
 	close(fd1);
 }
 
+struct fault_args {
+	char *addr;
+	char value;
+};
+
+static void *fault_thread_fn(void *arg)
+{
+	struct fault_args *args = arg;
+
+	/* Trigger page fault */
+	args->value = *args->addr;
+	return NULL;
+}
+
+static void test_uffd_minor(int fd, size_t total_size)
+{
+	struct uffdio_register uffd_reg;
+	struct uffdio_continue uffd_cont;
+	struct uffd_msg msg;
+	struct fault_args args;
+	pthread_t fault_thread;
+	void *mem, *mem_nofault, *buf = NULL;
+	int uffd, ret;
+	off_t offset = page_size;
+	void *fault_addr;
+	const char test_val = 0xcd;
+
+	ret = posix_memalign(&buf, page_size, total_size);
+	TEST_ASSERT_EQ(ret, 0);
+	memset(buf, test_val, total_size);
+
+	uffd = syscall(__NR_userfaultfd, O_CLOEXEC);
+	TEST_ASSERT(uffd != -1, "userfaultfd creation should succeed");
+
+	struct uffdio_api uffdio_api = {
+		.api = UFFD_API,
+		.features = 0,
+	};
+	ret = ioctl(uffd, UFFDIO_API, &uffdio_api);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_API) should succeed");
+
+	/* Map the guest_memfd twice: once with UFFD registered, once without */
+	mem = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem != MAP_FAILED, "mmap should succeed");
+
+	mem_nofault = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem_nofault != MAP_FAILED, "mmap should succeed");
+
+	/* Register UFFD_MINOR on the first mapping */
+	uffd_reg.range.start = (unsigned long)mem;
+	uffd_reg.range.len = total_size;
+	uffd_reg.mode = UFFDIO_REGISTER_MODE_MINOR;
+	ret = ioctl(uffd, UFFDIO_REGISTER, &uffd_reg);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_REGISTER) should succeed");
+
+	/*
+	 * Populate the page in the page cache first via mem_nofault.
+	 * This is required for UFFD_MINOR - the page must exist in the cache.
+	 * Write test data to the page.
+	 */
+	memcpy(mem_nofault + offset, buf + offset, page_size);
+
+	/*
+	 * Now access the same page via mem (which has UFFD_MINOR registered).
+	 * Since the page exists in the cache, this should trigger UFFD_MINOR.
+	 */
+	fault_addr = mem + offset;
+	args.addr = fault_addr;
+
+	ret = pthread_create(&fault_thread, NULL, fault_thread_fn, &args);
+	TEST_ASSERT(ret == 0, "pthread_create should succeed");
+
+	ret = read(uffd, &msg, sizeof(msg));
+	TEST_ASSERT(ret != -1, "read from userfaultfd should succeed");
+	TEST_ASSERT(msg.event == UFFD_EVENT_PAGEFAULT, "event type should be pagefault");
+	TEST_ASSERT((void *)(msg.arg.pagefault.address & ~(page_size - 1)) == fault_addr,
+		    "pagefault should occur at expected address");
+	TEST_ASSERT(msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR,
+		    "pagefault should be minor fault");
+
+	/* Resolve the minor fault with UFFDIO_CONTINUE */
+	uffd_cont.range.start = (unsigned long)fault_addr;
+	uffd_cont.range.len = page_size;
+	uffd_cont.mode = 0;
+	ret = ioctl(uffd, UFFDIO_CONTINUE, &uffd_cont);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_CONTINUE) should succeed");
+
+	/* Wait for the faulting thread to complete */
+	ret = pthread_join(fault_thread, NULL);
+	TEST_ASSERT(ret == 0, "pthread_join should succeed");
+
+	/* Verify the thread read the correct value */
+	TEST_ASSERT(args.value == test_val,
+		    "memory should contain the value that was written");
+	TEST_ASSERT(*(char *)(mem + offset) == test_val,
+		    "no further fault is expected");
+
+	ret = munmap(mem_nofault, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+
+	ret = munmap(mem, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+	free(buf);
+	close(uffd);
+}
+
 static void test_guest_memfd_flags(struct kvm_vm *vm)
 {
 	uint64_t valid_flags = vm_check_cap(vm, KVM_CAP_GUEST_MEMFD_FLAGS);
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)
 	gmem_test(file_size, vm, flags);
 	gmem_test(fallocate, vm, flags);
 	gmem_test(invalid_punch_hole, vm, flags);
+
+	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED)
+		gmem_test(uffd_minor, vm, flags);
 }
 
 static void test_guest_memfd(unsigned long vm_type)
Signature · 1 line
2.51.0
author maintainer
#38 lore raw

PATCH v2 15/15 KVM: selftests: test userfaultfd missing for guest_memfd

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.

---
 .../testing/selftests/kvm/guest_memfd_test.c  | 80 ++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
index 7612819e340a..f77e70d22175 100644
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)
 	close(uffd);
 }
 
+static void test_uffd_missing(int fd, size_t total_size)
+{
+	struct uffdio_register uffd_reg;
+	struct uffdio_copy uffd_copy;
+	struct uffd_msg msg;
+	struct fault_args args;
+	pthread_t fault_thread;
+	void *mem, *buf = NULL;
+	int uffd, ret;
+	off_t offset = page_size;
+	void *fault_addr;
+	const char test_val = 0xab;
+
+	ret = posix_memalign(&buf, page_size, total_size);
+	TEST_ASSERT_EQ(ret, 0);
+	memset(buf, test_val, total_size);
+
+	uffd = syscall(__NR_userfaultfd, O_CLOEXEC);
+	TEST_ASSERT(uffd != -1, "userfaultfd creation should succeed");
+
+	struct uffdio_api uffdio_api = {
+		.api = UFFD_API,
+		.features = 0,
+	};
+	ret = ioctl(uffd, UFFDIO_API, &uffdio_api);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_API) should succeed");
+
+	mem = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem != MAP_FAILED, "mmap should succeed");
+
+	uffd_reg.range.start = (unsigned long)mem;
+	uffd_reg.range.len = total_size;
+	uffd_reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+	ret = ioctl(uffd, UFFDIO_REGISTER, &uffd_reg);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_REGISTER) should succeed");
+
+	fault_addr = mem + offset;
+	args.addr = fault_addr;
+
+	ret = pthread_create(&fault_thread, NULL, fault_thread_fn, &args);
+	TEST_ASSERT(ret == 0, "pthread_create should succeed");
+
+	ret = read(uffd, &msg, sizeof(msg));
+	TEST_ASSERT(ret != -1, "read from userfaultfd should succeed");
+	TEST_ASSERT(msg.event == UFFD_EVENT_PAGEFAULT, "event type should be pagefault");
+	TEST_ASSERT((void *)(msg.arg.pagefault.address & ~(page_size - 1)) == fault_addr,
+		    "pagefault should occur at expected address");
+	TEST_ASSERT(!(msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP),
+		    "pagefault should not be write-protect");
+
+	uffd_copy.dst = (unsigned long)fault_addr;
+	uffd_copy.src = (unsigned long)(buf + offset);
+	uffd_copy.len = page_size;
+	uffd_copy.mode = 0;
+	ret = ioctl(uffd, UFFDIO_COPY, &uffd_copy);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_COPY) should succeed");
+
+	/* Wait for the faulting thread to complete - this provides the memory barrier */
+	ret = pthread_join(fault_thread, NULL);
+	TEST_ASSERT(ret == 0, "pthread_join should succeed");
+
+	/*
+	 * Now it's safe to check args.value - the thread has completed
+	 * and memory is synchronized
+	 */
+	TEST_ASSERT(args.value == test_val,
+		    "memory should contain the value that was copied");
+	TEST_ASSERT(*(char *)(mem + offset) == test_val,
+		    "no further fault is expected");
+
+	ret = munmap(mem, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+	free(buf);
+	close(uffd);
+}
+
 static void test_guest_memfd_flags(struct kvm_vm *vm)
 {
 	uint64_t valid_flags = vm_check_cap(vm, KVM_CAP_GUEST_MEMFD_FLAGS);
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)
 	gmem_test(fallocate, vm, flags);
 	gmem_test(invalid_punch_hole, vm, flags);
 
-	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED)
+	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED) {
 		gmem_test(uffd_minor, vm, flags);
+		gmem_test(uffd_missing, vm, flags);
+	}
 }
 
 static void test_guest_memfd(unsigned long vm_type)
Signature · 1 line
2.51.0
maintainer
#39 lore raw
In reply to Mike Rapoport
Quoted 1 line · On Fri, 6 Mar 2026 19:18:00 +0200 Mike Rapoport <[email protected]> wrote:

These patches enable support for userfaultfd in guest_memfd.

Thanks, Mike. I'll get this into the pipeline for testing.

Review is a little patchy at this time but hopefully that will improve over the next few weeks.

maintainer
#40 lore raw
In reply to Mike Rapoport
Quoted 1 line · On Fri, 6 Mar 2026 19:18:00 +0200 Mike Rapoport <[email protected]> wrote:

These patches enable support for userfaultfd in guest_memfd.

Review of this series is very thin at this time so can I please ask potential reviewers to spend some time with it?

To create additional time for review I have relocated this series to the tail of the mm-unstable queue. That probably means that movement into the mm-stable branch and upstreaming will occur in the second week of the upcoming (7.1-rc1) merge window.

Thanks.