Skip to contentSkip to conversation

mm, kvm: allow uffd support in guest_memfd

PATCH In review Mike Rapoport 7 participants 30 messages Apr 2 – May 19

Subsystems (10)
Memory management VFS KVM Kselftest +6 more subsystems
lore
Share & tools
lore

Mike Rapoport

author maintainer
#0 lore raw

v4 cover mm, kvm: allow uffd support in guest_memfd

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

These patches enable support for userfaultfd in guest_memfd.

As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.

In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.

Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
The handling of UFFDIO_MISSING in guest_memfd requires ability to remove a folio from page cache, the best way I could find was exporting filemap_remove_folio() to KVM.

I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.

Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.

v4 changes:
* fix lock leak not only in patch 4, but also in patch 2, thanks Harry

v3: https://lore.kernel.org/all/[email protected]
* add fixes from Harry and Andrei
* fix handling of WP-only mode for WP_ASYNC contexts in vma_can_userfault()
* address David's comments about mfill_get_pmd() and rename it to mfill_establish_pmd()
* add VM_WARN()s for unsupported operations (James)
* update comments using James' suggestions

v2: https://lore.kernel.org/all/[email protected]
* instead of returning uffd-specific values from ->fault() handlers add __do_userfault() helper to resolve user faults in __do_fault()
* address comments from Peter
* rebased on v7.0-c1

RFC: https://lore.kernel.org/all/[email protected]

Mike Rapoport (Microsoft) (11):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_establish_pmd() helper
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in
mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic(): remove retry logic

Nikita Kalyazin (3):
KVM: guest_memfd: implement userfaultfd operations
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd

Peter Xu (1):
mm: generalize handling of userfaults in __do_fault()

 include/linux/mm.h                            |   5 +
 include/linux/shmem_fs.h                      |  14 -
 include/linux/userfaultfd_k.h                 |  73 +-
 mm/filemap.c                                  |   1 +
 mm/hugetlb.c                                  |  15 +
 mm/memory.c                                   |  43 ++
 mm/shmem.c                                    | 188 ++---
 mm/userfaultfd.c                              | 694 ++++++++++--------
 .../testing/selftests/kvm/guest_memfd_test.c  | 191 +++++
 virt/kvm/guest_memfd.c                        |  84 ++-
 10 files changed, 860 insertions(+), 448 deletions(-)

base-commit: c369299895a591d96745d6492d4888259b004a9e --
2.53.0

author maintainer
#1 lore raw

PATCH v4 1/15 userfaultfd: introduce mfill_copy_folio_locked() helper

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().

This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.

No functional change.

---
 mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
index 927086bb4a3c..32637d557c95 100644
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,
 	return ret;
 }
 
+static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
+{
+	void *kaddr;
+	int ret;
+
+	kaddr = kmap_local_folio(folio, 0);
+	/*
+	 * The read mmap_lock is held here.  Despite the
+	 * mmap_lock being read recursive a deadlock is still
+	 * possible if a writer has taken a lock.  For example:
+	 *
+	 * process A thread 1 takes read lock on own mmap_lock
+	 * process A thread 2 calls mmap, blocks taking write lock
+	 * process B thread 1 takes page fault, read lock on own mmap lock
+	 * process B thread 2 calls mmap, blocks taking write lock
+	 * process A thread 1 blocks taking read lock on process B
+	 * process B thread 1 blocks taking read lock on process A
+	 *
+	 * Disable page faults to prevent potential deadlock
+	 * and retry the copy outside the mmap_lock.
+	 */
+	pagefault_disable();
+	ret = copy_from_user(kaddr, (const void __user *) src_addr,
+			     PAGE_SIZE);
+	pagefault_enable();
+	kunmap_local(kaddr);
+
+	if (ret)
+		return -EFAULT;
+
+	flush_dcache_folio(folio);
+	return ret;
+}
+
 static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 				 struct vm_area_struct *dst_vma,
 				 unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 				 uffd_flags_t flags,
 				 struct folio **foliop)
 {
-	void *kaddr;
 	int ret;
 	struct folio *folio;
 
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 		if (!folio)
 			goto out;
 
-		kaddr = kmap_local_folio(folio, 0);
-		/*
-		 * The read mmap_lock is held here.  Despite the
-		 * mmap_lock being read recursive a deadlock is still
-		 * possible if a writer has taken a lock.  For example:
-		 *
-		 * process A thread 1 takes read lock on own mmap_lock
-		 * process A thread 2 calls mmap, blocks taking write lock
-		 * process B thread 1 takes page fault, read lock on own mmap lock
-		 * process B thread 2 calls mmap, blocks taking write lock
-		 * process A thread 1 blocks taking read lock on process B
-		 * process B thread 1 blocks taking read lock on process A
-		 *
-		 * Disable page faults to prevent potential deadlock
-		 * and retry the copy outside the mmap_lock.
-		 */
-		pagefault_disable();
-		ret = copy_from_user(kaddr, (const void __user *) src_addr,
-				     PAGE_SIZE);
-		pagefault_enable();
-		kunmap_local(kaddr);
+		ret = mfill_copy_folio_locked(folio, src_addr);
 
 		/* fallback to copy_from_user outside mmap_lock */
 		if (unlikely(ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 			/* don't free the page */
 			goto out;
 		}
-
-		flush_dcache_folio(folio);
 	} else {
 		folio = *foliop;
 		*foliop = NULL;
Signature · 1 line
2.53.0
author maintainer
#2 lore raw

PATCH v4 2/15 userfaultfd: introduce struct mfill_state

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

mfill_atomic() passes a lot of parameters down to its callees.

Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.

Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.

The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.

[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap() WARN]

---
 mm/userfaultfd.c | 147 ++++++++++++++++++++++++++---------------------
 1 file changed, 81 insertions(+), 66 deletions(-)
mm/userfaultfd.c +81 -66
index 32637d557c95..c4074b6f4aca 100644
@@ -20,6 +20,20 @@
 #include "internal.h"
 #include "swap.h"
 
+struct mfill_state {
+	struct userfaultfd_ctx *ctx;
+	unsigned long src_start;
+	unsigned long dst_start;
+	unsigned long len;
+	uffd_flags_t flags;
+
+	struct vm_area_struct *vma;
+	unsigned long src_addr;
+	unsigned long dst_addr;
+	struct folio *folio;
+	pmd_t *pmd;
+};
+
 static __always_inline
 bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end)
 {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
-				 struct vm_area_struct *dst_vma,
-				 unsigned long dst_addr,
-				 unsigned long src_addr,
-				 uffd_flags_t flags,
-				 struct folio **foliop)
+static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-	int ret;
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
+	unsigned long src_addr = state->src_addr;
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
+	int ret;
 
-	if (!*foliop) {
+	if (!state->folio) {
 		ret = -ENOMEM;
 		folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma,
 					dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 		/* fallback to copy_from_user outside mmap_lock */
 		if (unlikely(ret)) {
 			ret = -ENOENT;
-			*foliop = folio;
+			state->folio = folio;
 			/* don't free the page */
 			goto out;
 		}
 	} else {
-		folio = *foliop;
-		*foliop = NULL;
+		folio = state->folio;
+		state->folio = NULL;
 	}
 
 	/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,
 	return ret;
 }
 
-static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
-				     struct vm_area_struct *dst_vma,
-				     unsigned long dst_addr)
+static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 {
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
+	pmd_t *dst_pmd = state->pmd;
 	pte_t _dst_pte, *dst_pte;
 	spinlock_t *ptl;
 	int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
 }
 
 /* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */
-static int mfill_atomic_pte_continue(pmd_t *dst_pmd,
-				     struct vm_area_struct *dst_vma,
-				     unsigned long dst_addr,
-				     uffd_flags_t flags)
+static int mfill_atomic_pte_continue(struct mfill_state *state)
 {
-	struct inode *inode = file_inode(dst_vma->vm_file);
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long dst_addr = state->dst_addr;
 	pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
+	struct inode *inode = file_inode(dst_vma->vm_file);
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
 	struct page *page;
 	int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd,
 }
 
 /* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
-static int mfill_atomic_pte_poison(pmd_t *dst_pmd,
-				   struct vm_area_struct *dst_vma,
-				   unsigned long dst_addr,
-				   uffd_flags_t flags)
+static int mfill_atomic_pte_poison(struct mfill_state *state)
 {
-	int ret;
+	struct vm_area_struct *dst_vma = state->vma;
 	struct mm_struct *dst_mm = dst_vma->vm_mm;
+	unsigned long dst_addr = state->dst_addr;
+	pmd_t *dst_pmd = state->pmd;
 	pte_t _dst_pte, *dst_pte;
 	spinlock_t *ptl;
+	int ret;
 
 	_dst_pte = make_pte_marker(PTE_MARKER_POISONED);
 	ret = -EAGAIN;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx,
 				    uffd_flags_t flags);
 #endif /* CONFIG_HUGETLB_PAGE */
 
-static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,
-						struct vm_area_struct *dst_vma,
-						unsigned long dst_addr,
-						unsigned long src_addr,
-						uffd_flags_t flags,
-						struct folio **foliop)
+static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 {
+	struct vm_area_struct *dst_vma = state->vma;
+	unsigned long src_addr = state->src_addr;
+	unsigned long dst_addr = state->dst_addr;
+	struct folio **foliop = &state->folio;
+	uffd_flags_t flags = state->flags;
+	pmd_t *dst_pmd = state->pmd;
 	ssize_t err;
 
-	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) {
-		return mfill_atomic_pte_continue(dst_pmd, dst_vma,
-						 dst_addr, flags);
-	} else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) {
-		return mfill_atomic_pte_poison(dst_pmd, dst_vma,
-					       dst_addr, flags);
-	}
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+		return mfill_atomic_pte_continue(state);
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
+		return mfill_atomic_pte_poison(state);
 
 	/*
 	 * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,
 	 */
 	if (!(dst_vma->vm_flags & VM_SHARED)) {
 		if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-			err = mfill_atomic_pte_copy(dst_pmd, dst_vma,
-						    dst_addr, src_addr,
-						    flags, foliop);
+			err = mfill_atomic_pte_copy(state);
 		else
-			err = mfill_atomic_pte_zeropage(dst_pmd,
-						 dst_vma, dst_addr);
+			err = mfill_atomic_pte_zeropage(state);
 	} else {
 		err = shmem_mfill_atomic_pte(dst_pmd, dst_vma,
 					     dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 					    unsigned long len,
 					    uffd_flags_t flags)
 {
+	struct mfill_state state = (struct mfill_state){
+		.ctx = ctx,
+		.dst_start = dst_start,
+		.src_start = src_start,
+		.flags = flags,
+		.len = len,
+		.src_addr = src_start,
+		.dst_addr = dst_start,
+	};
 	struct mm_struct *dst_mm = ctx->mm;
 	struct vm_area_struct *dst_vma;
+	long copied = 0;
 	ssize_t err;
 	pmd_t *dst_pmd;
-	unsigned long src_addr, dst_addr;
-	long copied;
-	struct folio *folio;
 
 	/*
 	 * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(src_start + len <= src_start);
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
-	src_addr = src_start;
-	dst_addr = dst_start;
-	copied = 0;
-	folio = NULL;
 retry:
 	/*
 	 * Make sure the vma is not shared, that the dst range is
@@ -750,6 +764,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		err = PTR_ERR(dst_vma);
 		goto out;
 	}
+	state.vma = dst_vma;
 
 	/*
 	 * If memory mappings are changing because of non-cooperative
@@ -790,12 +805,12 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
 		goto out_unlock;
 
-	while (src_addr < src_start + len) {
-		pmd_t dst_pmdval;
+	while (state.src_addr < src_start + len) {
+		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
-		VM_WARN_ON_ONCE(dst_addr >= dst_start + len);
+		pmd_t dst_pmdval;
 
-		dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
 		if (unlikely(!dst_pmd)) {
 			err = -ENOMEM;
 			break;
@@ -827,34 +842,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		 * tables under us; pte_offset_map_lock() will deal with that.
 		 */
 
-		err = mfill_atomic_pte(dst_pmd, dst_vma, dst_addr,
-				       src_addr, flags, &folio);
+		state.pmd = dst_pmd;
+		err = mfill_atomic_pte(&state);
 		cond_resched();
 
 		if (unlikely(err == -ENOENT)) {
 			void *kaddr;
 
 			up_read(&ctx->map_changing_lock);
-			uffd_mfill_unlock(dst_vma);
-			VM_WARN_ON_ONCE(!folio);
+			uffd_mfill_unlock(state.vma);
+			VM_WARN_ON_ONCE(!state.folio);
 
-			kaddr = kmap_local_folio(folio, 0);
+			kaddr = kmap_local_folio(state.folio, 0);
 			err = copy_from_user(kaddr,
-					     (const void __user *) src_addr,
+					     (const void __user *)state.src_addr,
 					     PAGE_SIZE);
 			kunmap_local(kaddr);
 			if (unlikely(err)) {
 				err = -EFAULT;
 				goto out;
 			}
-			flush_dcache_folio(folio);
+			flush_dcache_folio(state.folio);
 			goto retry;
 		} else
-			VM_WARN_ON_ONCE(folio);
+			VM_WARN_ON_ONCE(state.folio);
 
 		if (!err) {
-			dst_addr += PAGE_SIZE;
-			src_addr += PAGE_SIZE;
+			state.dst_addr += PAGE_SIZE;
+			state.src_addr += PAGE_SIZE;
 			copied += PAGE_SIZE;
 
 			if (fatal_signal_pending(current))
@@ -866,10 +881,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
 out_unlock:
 	up_read(&ctx->map_changing_lock);
-	uffd_mfill_unlock(dst_vma);
+	uffd_mfill_unlock(state.vma);
 out:
-	if (folio)
-		folio_put(folio);
+	if (state.folio)
+		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);
Signature · 1 line
2.53.0
Reviewed-by on v4 2/15
#3 lore raw

Re: [PATCH v4 02/15] userfaultfd: introduce struct mfill_state

Quoted 15 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

mfill_atomic() passes a lot of parameters down to its callees.

Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.

Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.

The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.

Quoted 6 lines

[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap() WARN]

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#4 lore raw

PATCH v4 3/15 userfaultfd: introduce mfill_establish_pmd() helper

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.

Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().

---
 mm/userfaultfd.c | 102 ++++++++++++++++++++++++-----------------------
 1 file changed, 52 insertions(+), 50 deletions(-)
mm/userfaultfd.c +52 -50
index c4074b6f4aca..f653fb3f74df 100644
@@ -157,6 +157,56 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+
+	pgd = pgd_offset(mm, address);
+	p4d = p4d_alloc(mm, pgd, address);
+	if (!p4d)
+		return NULL;
+	pud = pud_alloc(mm, p4d, address);
+	if (!pud)
+		return NULL;
+	/*
+	 * Note that we didn't run this because the pmd was
+	 * missing, the *pmd may be already established and in
+	 * turn it may also be a trans_huge_pmd.
+	 */
+	return pmd_alloc(mm, pud, address);
+}
+
+static int mfill_establish_pmd(struct mfill_state *state)
+{
+	struct mm_struct *dst_mm = state->ctx->mm;
+	pmd_t *dst_pmd, dst_pmdval;
+
+	dst_pmd = mm_alloc_pmd(dst_mm, state->dst_addr);
+	if (unlikely(!dst_pmd))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	if (unlikely(pmd_none(dst_pmdval)) &&
+	    unlikely(__pte_alloc(dst_mm, dst_pmd)))
+		return -ENOMEM;
+
+	dst_pmdval = pmdp_get_lockless(dst_pmd);
+	/*
+	 * If the dst_pmd is THP don't override it and just be strict.
+	 * (This includes the case where the PMD used to be THP and
+	 * changed back to none after __pte_alloc().)
+	 */
+	if (unlikely(!pmd_present(dst_pmdval) || pmd_leaf(dst_pmdval)))
+		return -EEXIST;
+	if (unlikely(pmd_bad(dst_pmdval)))
+		return -EFAULT;
+
+	state->pmd = dst_pmd;
+	return 0;
+}
+
 /* Check if dst_addr is outside of file's size. Must be called with ptl held. */
 static bool mfill_file_over_size(struct vm_area_struct *dst_vma,
 				 unsigned long dst_addr)
@@ -489,27 +539,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)
 	return ret;
 }
 
-static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
-{
-	pgd_t *pgd;
-	p4d_t *p4d;
-	pud_t *pud;
-
-	pgd = pgd_offset(mm, address);
-	p4d = p4d_alloc(mm, pgd, address);
-	if (!p4d)
-		return NULL;
-	pud = pud_alloc(mm, p4d, address);
-	if (!pud)
-		return NULL;
-	/*
-	 * Note that we didn't run this because the pmd was
-	 * missing, the *pmd may be already established and in
-	 * turn it may also be a trans_huge_pmd.
-	 */
-	return pmd_alloc(mm, pud, address);
-}
-
 #ifdef CONFIG_HUGETLB_PAGE
 /*
  * mfill_atomic processing for HUGETLB vmas.  Note that this routine is
@@ -742,7 +771,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	struct vm_area_struct *dst_vma;
 	long copied = 0;
 	ssize_t err;
-	pmd_t *dst_pmd;
 
 	/*
 	 * Sanitize the command parameters:
@@ -808,41 +836,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
-		pmd_t dst_pmdval;
-
-		dst_pmd = mm_alloc_pmd(dst_mm, state.dst_addr);
-		if (unlikely(!dst_pmd)) {
-			err = -ENOMEM;
+		err = mfill_establish_pmd(&state);
+		if (err)
 			break;
-		}
 
-		dst_pmdval = pmdp_get_lockless(dst_pmd);
-		if (unlikely(pmd_none(dst_pmdval)) &&
-		    unlikely(__pte_alloc(dst_mm, dst_pmd))) {
-			err = -ENOMEM;
-			break;
-		}
-		dst_pmdval = pmdp_get_lockless(dst_pmd);
-		/*
-		 * If the dst_pmd is THP don't override it and just be strict.
-		 * (This includes the case where the PMD used to be THP and
-		 * changed back to none after __pte_alloc().)
-		 */
-		if (unlikely(!pmd_present(dst_pmdval) ||
-				pmd_trans_huge(dst_pmdval))) {
-			err = -EEXIST;
-			break;
-		}
-		if (unlikely(pmd_bad(dst_pmdval))) {
-			err = -EFAULT;
-			break;
-		}
 		/*
 		 * For shmem mappings, khugepaged is allowed to remove page
 		 * tables under us; pte_offset_map_lock() will deal with that.
 		 */
 
-		state.pmd = dst_pmd;
 		err = mfill_atomic_pte(&state);
 		cond_resched();
 
Signature · 1 line
2.53.0
author maintainer
#5 lore raw

PATCH v4 4/15 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.

This function will be used later during refactoring of mfill_atomic_pte_copy().

Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.

---
 mm/userfaultfd.c | 125 ++++++++++++++++++++++++++++-------------------
 1 file changed, 75 insertions(+), 50 deletions(-)
mm/userfaultfd.c +75 -50
index f653fb3f74df..c6a38db45343 100644
@@ -157,6 +157,75 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)
 }
 #endif
 
+static void mfill_put_vma(struct mfill_state *state)
+{
+	if (!state->vma)
+		return;
+
+	up_read(&state->ctx->map_changing_lock);
+	uffd_mfill_unlock(state->vma);
+	state->vma = NULL;
+}
+
+static int mfill_get_vma(struct mfill_state *state)
+{
+	struct userfaultfd_ctx *ctx = state->ctx;
+	uffd_flags_t flags = state->flags;
+	struct vm_area_struct *dst_vma;
+	int err;
+
+	/*
+	 * Make sure the vma is not shared, that the dst range is
+	 * both valid and fully within a single existing vma.
+	 */
+	dst_vma = uffd_mfill_lock(ctx->mm, state->dst_start, state->len);
+	if (IS_ERR(dst_vma))
+		return PTR_ERR(dst_vma);
+
+	/*
+	 * If memory mappings are changing because of non-cooperative
+	 * operation (e.g. mremap) running in parallel, bail out and
+	 * request the user to retry later
+	 */
+	down_read(&ctx->map_changing_lock);
+	state->vma = dst_vma;
+	err = -EAGAIN;
+	if (atomic_read(&ctx->mmap_changing))
+		goto out_unlock;
+
+	err = -EINVAL;
+
+	/*
+	 * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
+	 * it will overwrite vm_ops, so vma_is_anonymous must return false.
+	 */
+	if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
+	    dst_vma->vm_flags & VM_SHARED))
+		goto out_unlock;
+
+	/*
+	 * validate 'mode' now that we know the dst_vma: don't allow
+	 * a wrprotect copy if the userfaultfd didn't register as WP.
+	 */
+	if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))
+		goto out_unlock;
+
+	if (is_vm_hugetlb_page(dst_vma))
+		return 0;
+
+	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+		goto out_unlock;
+	if (!vma_is_shmem(dst_vma) &&
+	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+		goto out_unlock;
+
+	return 0;
+
+out_unlock:
+	mfill_put_vma(state);
+	return err;
+}
+
 static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
 {
 	pgd_t *pgd;
@@ -767,8 +836,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		.src_addr = src_start,
 		.dst_addr = dst_start,
 	};
-	struct mm_struct *dst_mm = ctx->mm;
-	struct vm_area_struct *dst_vma;
 	long copied = 0;
 	ssize_t err;
 
@@ -783,56 +850,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
 retry:
-	/*
-	 * Make sure the vma is not shared, that the dst range is
-	 * both valid and fully within a single existing vma.
-	 */
-	dst_vma = uffd_mfill_lock(dst_mm, dst_start, len);
-	if (IS_ERR(dst_vma)) {
-		err = PTR_ERR(dst_vma);
+	err = mfill_get_vma(&state);
+	if (err)
 		goto out;
-	}
-	state.vma = dst_vma;
-
-	/*
-	 * If memory mappings are changing because of non-cooperative
-	 * operation (e.g. mremap) running in parallel, bail out and
-	 * request the user to retry later
-	 */
-	down_read(&ctx->map_changing_lock);
-	err = -EAGAIN;
-	if (atomic_read(&ctx->mmap_changing))
-		goto out_unlock;
-
-	err = -EINVAL;
-	/*
-	 * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
-	 * it will overwrite vm_ops, so vma_is_anonymous must return false.
-	 */
-	if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
-	    dst_vma->vm_flags & VM_SHARED))
-		goto out_unlock;
-
-	/*
-	 * validate 'mode' now that we know the dst_vma: don't allow
-	 * a wrprotect copy if the userfaultfd didn't register as WP.
-	 */
-	if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))
-		goto out_unlock;
 
 	/*
 	 * If this is a HUGETLB vma, pass off to appropriate routine
 	 */
-	if (is_vm_hugetlb_page(dst_vma))
-		return  mfill_atomic_hugetlb(ctx, dst_vma, dst_start,
+	if (is_vm_hugetlb_page(state.vma))
+		return  mfill_atomic_hugetlb(ctx, state.vma, dst_start,
 					     src_start, len, flags);
 
-	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
-		goto out_unlock;
-	if (!vma_is_shmem(dst_vma) &&
-	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
-		goto out_unlock;
-
 	while (state.src_addr < src_start + len) {
 		VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
 
@@ -851,8 +879,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		if (unlikely(err == -ENOENT)) {
 			void *kaddr;
 
-			up_read(&ctx->map_changing_lock);
-			uffd_mfill_unlock(state.vma);
+			mfill_put_vma(&state);
 			VM_WARN_ON_ONCE(!state.folio);
 
 			kaddr = kmap_local_folio(state.folio, 0);
@@ -881,9 +908,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 			break;
 	}
 
-out_unlock:
-	up_read(&ctx->map_changing_lock);
-	uffd_mfill_unlock(state.vma);
+	mfill_put_vma(&state);
 out:
 	if (state.folio)
 		folio_put(state.folio);
Signature · 1 line
2.53.0
Reviewed-by on v4 4/15
#6 lore raw

Re: [PATCH v4 04/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()

Quoted 8 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.

This function will be used later during refactoring of mfill_atomic_pte_copy().

Quoted 8 lines

Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#7 lore raw

PATCH v4 5/15 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).

In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.

Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.

As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.

---
 mm/userfaultfd.c | 75 ++++++++++++++++++++++++++++++++----------------
 1 file changed, 51 insertions(+), 24 deletions(-)
mm/userfaultfd.c +51 -24
index c6a38db45343..82e1a3255e1e 100644
@@ -405,35 +405,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
+static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+{
+	unsigned long src_addr = state->src_addr;
+	void *kaddr;
+	int err;
+
+	/* retry copying with mm_lock dropped */
+	mfill_put_vma(state);
+
+	kaddr = kmap_local_folio(folio, 0);
+	err = copy_from_user(kaddr, (const void __user *) src_addr, PAGE_SIZE);
+	kunmap_local(kaddr);
+	if (unlikely(err))
+		return -EFAULT;
+
+	flush_dcache_folio(folio);
+
+	/* reget VMA and PMD, they could change underneath us */
+	err = mfill_get_vma(state);
+	if (err)
+		return err;
+
+	err = mfill_establish_pmd(state);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-	struct vm_area_struct *dst_vma = state->vma;
 	unsigned long dst_addr = state->dst_addr;
 	unsigned long src_addr = state->src_addr;
 	uffd_flags_t flags = state->flags;
-	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
 	int ret;
 
-	if (!state->folio) {
-		ret = -ENOMEM;
-		folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma,
-					dst_addr);
-		if (!folio)
-			goto out;
+	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, state->vma, dst_addr);
+	if (!folio)
+		return -ENOMEM;
 
-		ret = mfill_copy_folio_locked(folio, src_addr);
+	ret = -ENOMEM;
+	if (mem_cgroup_charge(folio, state->vma->vm_mm, GFP_KERNEL))
+		goto out_release;
 
-		/* fallback to copy_from_user outside mmap_lock */
-		if (unlikely(ret)) {
-			ret = -ENOENT;
-			state->folio = folio;
-			/* don't free the page */
-			goto out;
-		}
-	} else {
-		folio = state->folio;
-		state->folio = NULL;
+	ret = mfill_copy_folio_locked(folio, src_addr);
+	if (unlikely(ret)) {
+		/*
+		 * Fallback to copy_from_user outside mmap_lock.
+		 * If retry is successful, mfill_copy_folio_locked() returns
+		 * with locks retaken by mfill_get_vma().
+		 * If there was an error, we must mfill_put_vma() anyway and it
+		 * will take care of unlocking if needed.
+		 */
+		ret = mfill_copy_folio_retry(state, folio);
+		if (ret)
+			goto out_release;
 	}
 
 	/*
@@ -443,17 +471,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)
 	 */
 	__folio_mark_uptodate(folio);
 
-	ret = -ENOMEM;
-	if (mem_cgroup_charge(folio, dst_vma->vm_mm, GFP_KERNEL))
-		goto out_release;
-
-	ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
+	ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr,
 				       &folio->page, true, flags);
 	if (ret)
 		goto out_release;
 out:
 	return ret;
 out_release:
+	/* Don't return -ENOENT so that our caller won't retry */
+	if (ret == -ENOENT)
+		ret = -EFAULT;
 	folio_put(folio);
 	goto out;
 }
Signature · 1 line
2.53.0
Reviewed-by on v4 5/15
#8 lore raw

Re: [PATCH v4 05/15] userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()

Quoted 21 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).

In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.

Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.

As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.

Quoted 6 lines

---

Looks good to me,

Quoted 39 lines
 mm/userfaultfd.c | 75 ++++++++++++++++++++++++++++++++----------------
 1 file changed, 51 insertions(+), 24 deletions(-)
mm/userfaultfd.c +6 -15
index c6a38db45343..82e1a3255e1e 100644
@@ -405,35 +405,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 static int mfill_atomic_pte_copy(struct mfill_state *state)
 {
-	struct vm_area_struct *dst_vma = state->vma;
 	unsigned long dst_addr = state->dst_addr;
 	unsigned long src_addr = state->src_addr;
 	uffd_flags_t flags = state->flags;
-	pmd_t *dst_pmd = state->pmd;
 	struct folio *folio;
 	int ret;
 
-	if (!state->folio) {
-		ret = -ENOMEM;
-		folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma,
-					dst_addr);
-		if (!folio)
-			goto out;
+	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, state->vma, dst_addr);
+	if (!folio)
+		return -ENOMEM;
 
-		ret = mfill_copy_folio_locked(folio, src_addr);
+	ret = -ENOMEM;
+	if (mem_cgroup_charge(folio, state->vma->vm_mm, GFP_KERNEL))
+		goto out_release;
 
-		/* fallback to copy_from_user outside mmap_lock */
-		if (unlikely(ret)) {
-			ret = -ENOENT;
-			state->folio = folio;
-			/* don't free the page */
-			goto out;

mm/userfaultfd.c static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) · line 426

Quoted 10 lines
mm/userfaultfd.c +6 -4
-		}
-	} else {
-		folio = state->folio;
-		state->folio = NULL;
+	ret = mfill_copy_folio_locked(folio, src_addr);
+	if (unlikely(ret)) {
+		/*
+		 * Fallback to copy_from_user outside mmap_lock.
+		 * If retry is successful, mfill_copy_folio_locked() returns
+		 * with locks retaken by mfill_get_vma().

nit: mfill_copy_folio_locked() -> mfill_copy_folio_retry();

Quoted 9 lines
mm/userfaultfd.c +6 -0
+		 * If there was an error, we must mfill_put_vma() anyway and it
+		 * will take care of unlocking if needed.
+		 */
+		ret = mfill_copy_folio_retry(state, folio);
+		if (ret)
+			goto out_release;
 	}
 
 	/*
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#9 lore raw

PATCH v4 6/15 userfaultfd: move vma_can_userfault out of line

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

vma_can_userfault() has grown pretty big and it's not called on performance critical path.

Move it out of line.

No functional changes.

---
 include/linux/userfaultfd_k.h | 35 ++---------------------------------
 mm/userfaultfd.c              | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
index fd5f42765497..a49cf750e803 100644
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
 	return vma->vm_flags & __VM_UFFD_FLAGS;
 }
 
-static inline bool vma_can_userfault(struct vm_area_struct *vma,
-				     vm_flags_t vm_flags,
-				     bool wp_async)
-{
-	vm_flags &= __VM_UFFD_FLAGS;
-
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
-	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
-		return false;
-
-	/*
-	 * If wp async enabled, and WP is the only mode enabled, allow any
-	 * memory type.
-	 */
-	if (wp_async && (vm_flags == VM_UFFD_WP))
-		return true;
-
-	/*
-	 * If user requested uffd-wp but not enabled pte markers for
-	 * uffd-wp, then shmem & hugetlbfs are not supported but only
-	 * anonymous.
-	 */
-	if (!uffd_supports_wp_marker() && (vm_flags & VM_UFFD_WP) &&
-	    !vma_is_anonymous(vma))
-		return false;
-
-	/* By default, allow any of anon|shmem|hugetlb */
-	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
-	    vma_is_shmem(vma);
-}
+bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
+		       bool wp_async);
 
 static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
 {
mm/userfaultfd.c +33 -0
index 82e1a3255e1e..d51e080e9216 100644
@@ -2018,6 +2018,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 	return moved ? moved : err;
 }
 
+bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
+		       bool wp_async)
+{
+	vm_flags &= __VM_UFFD_FLAGS;
+
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+
+	if ((vm_flags & VM_UFFD_MINOR) &&
+	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
+		return false;
+
+	/*
+	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * memory type.
+	 */
+	if (wp_async && (vm_flags == VM_UFFD_WP))
+		return true;
+
+	/*
+	 * If user requested uffd-wp but not enabled pte markers for
+	 * uffd-wp, then shmem & hugetlbfs are not supported but only
+	 * anonymous.
+	 */
+	if (!uffd_supports_wp_marker() && (vm_flags & VM_UFFD_WP) &&
+	    !vma_is_anonymous(vma))
+		return false;
+
+	/* By default, allow any of anon|shmem|hugetlb */
+	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
+	    vma_is_shmem(vma);
+}
+
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
 				     vm_flags_t vm_flags)
 {
Signature · 1 line
2.53.0
author maintainer
#10 lore raw

PATCH v4 7/15 userfaultfd: introduce vm_uffd_ops

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Current userfaultfd implementation works only with memory managed by core
MM: anonymous, shmem and hugetlb.

First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.

Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.

Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.

Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.

Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.

Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.

---
 include/linux/mm.h            |  5 ++++
 include/linux/userfaultfd_k.h |  6 +++++
 mm/hugetlb.c                  | 15 ++++++++++++
 mm/shmem.c                    | 15 ++++++++++++
 mm/userfaultfd.c              | 44 ++++++++++++++++++++++++-----------
 5 files changed, 72 insertions(+), 13 deletions(-)
include/linux/mm.h +5 -0
index abb4963c1f06..0fbeb8012983 100644
@@ -741,6 +741,8 @@ struct vm_fault {
 					 */
 };
 
+struct vm_uffd_ops;
+
 /*
  * These are the virtual MM functions - opening of an area, closing and
  * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -826,6 +828,9 @@ struct vm_operations_struct {
 	struct page *(*find_normal_page)(struct vm_area_struct *vma,
 					 unsigned long addr);
 #endif /* CONFIG_FIND_NORMAL_PAGE */
+#ifdef CONFIG_USERFAULTFD
+	const struct vm_uffd_ops *uffd_ops;
+#endif
 };
 
 #ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
index a49cf750e803..56e85ab166c7 100644
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {
 
 extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 
+/* VMA userfaultfd operations */
+struct vm_uffd_ops {
+	/* Checks if a VMA can support userfaultfd */
+	bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+};
+
 /* A combined operation mode + behavior flags. */
 typedef unsigned int __bitwise uffd_flags_t;
 
mm/hugetlb.c +15 -0
index 327eaa4074d3..530bc4337be6 100644
@@ -4818,6 +4818,18 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)
 	return 0;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static bool hugetlb_can_userfault(struct vm_area_struct *vma,
+				  vm_flags_t vm_flags)
+{
+	return true;
+}
+
+static const struct vm_uffd_ops hugetlb_uffd_ops = {
+	.can_userfault = hugetlb_can_userfault,
+};
+#endif
+
 /*
  * When a new function is introduced to vm_operations_struct and added
  * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4831,6 +4843,9 @@ const struct vm_operations_struct hugetlb_vm_ops = {
 	.close = hugetlb_vm_op_close,
 	.may_split = hugetlb_vm_op_split,
 	.pagesize = hugetlb_vm_op_pagesize,
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops = &hugetlb_uffd_ops,
+#endif
 };
 
 static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +15 -0
index b40f3cd48961..f2a25805b9bf 100644
@@ -3294,6 +3294,15 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
 	shmem_inode_unacct_blocks(inode, 1);
 	return ret;
 }
+
+static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	return true;
+}
+
+static const struct vm_uffd_ops shmem_uffd_ops = {
+	.can_userfault	= shmem_can_userfault,
+};
 #endif /* CONFIG_USERFAULTFD */
 
 #ifdef CONFIG_TMPFS
@@ -5313,6 +5322,9 @@ static const struct vm_operations_struct shmem_vm_ops = {
 	.set_policy     = shmem_set_policy,
 	.get_policy     = shmem_get_policy,
 #endif
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops	= &shmem_uffd_ops,
+#endif
 };
 
 static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5322,6 +5334,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = {
 	.set_policy     = shmem_set_policy,
 	.get_policy     = shmem_get_policy,
 #endif
+#ifdef CONFIG_USERFAULTFD
+	.uffd_ops	= &shmem_uffd_ops,
+#endif
 };
 
 int shmem_init_fs_context(struct fs_context *fc)
mm/userfaultfd.c +31 -13

71 lines not on this page · raw message

Signature · 1 line
2.53.0
#11 lore raw

Re: [PATCH v4 07/15] userfaultfd: introduce vm_uffd_ops

Hi Mike,

Quoted 20 lines · Mike Rapoport
mm/userfaultfd.c +3 -8
@@ -2021,34 +2040,33 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 		       bool wp_async)
 {
-	vm_flags &= __VM_UFFD_FLAGS;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
-	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
-		return false;
+	vm_flags &= __VM_UFFD_FLAGS;
 
 	/*
-	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * If WP is the only mode enabled and context is wp async, allow any
 	 * memory type.
 	 */

mm/userfaultfd.c ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2060

Quoted 10 lines
mm/userfaultfd.c +7 -0
 	if (wp_async && (vm_flags == VM_UFFD_WP))
 		return true;
 
+	/* For any other mode reject VMAs that don't implement vm_uffd_ops */
+	if (!ops)
+		return false;
+
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+

Doesn't moving the VM_DROPPABLE check after the wp_async early return make the issue David raised in [1] possible again (i.e. moving swapbacked pages into a droppable region using UFFDIO_MOVE)? With this change, VM_DROPPABLE pages can now be registered with uffd under wp_async, which allows UFFDIO_MOVE.

I think the VM_DROPPABLE check should go right before the wp_async check. And it would probably benefit from a comment...

[1] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Tal

author maintainer
#12 lore raw

Re: [PATCH v4 07/15] userfaultfd: introduce vm_uffd_ops

Quoted 40 lines · Tal Zussman

Hi Mike,

Quoted 20 lines · Mike Rapoport
(quoted diff) +3 -8
@@ -2021,34 +2040,33 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 		       bool wp_async)
 {
-	vm_flags &= __VM_UFFD_FLAGS;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
-	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
-		return false;
+	vm_flags &= __VM_UFFD_FLAGS;
 
 	/*
-	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * If WP is the only mode enabled and context is wp async, allow any
 	 * memory type.
 	 */

Review ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2060

Quoted 10 lines
(quoted diff) +7 -0
 	if (wp_async && (vm_flags == VM_UFFD_WP))
 		return true;
 
+	/* For any other mode reject VMAs that don't implement vm_uffd_ops */
+	if (!ops)
+		return false;
+
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+

Doesn't moving the VM_DROPPABLE check after the wp_async early return make the issue David raised in [1] possible again (i.e. moving swapbacked pages into a droppable region using UFFDIO_MOVE)? With this change, VM_DROPPABLE pages can now be registered with uffd under wp_async, which allows UFFDIO_MOVE.

Quoted 2 lines

I think the VM_DROPPABLE check should go right before the wp_async check. And it would probably benefit from a comment...

You are right.

Andrew, if it's not too late can you please add this fixup:

mm/userfaultfd.c +3 -3
index f1828106484b..75290483ffc7 100644
@@ -2019,6 +2019,9 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 {
 	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
 
+	if (vma->vm_flags & VM_DROPPABLE)
+		return false;
+
 	vm_flags &= __VM_UFFD_FLAGS;
 
 	/*
@@ -2032,9 +2035,6 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 	if (!ops)
 		return false;
 
-	if (vma->vm_flags & VM_DROPPABLE)
-		return false;
-
 	/*
 	 * If user requested uffd-wp but not enabled pte markers for
 	 * uffd-wp, then only anonymous memory is supported
Quoted 2 lines

Thanks,
Tal

Signature · 2 lines
Sincerely yours,
Mike.
author maintainer
#13 lore raw

PATCH v4 8/15 shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.

Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.

---
 include/linux/userfaultfd_k.h |  7 +++++++
 mm/shmem.c                    | 15 ++++++++++++++-
 mm/userfaultfd.c              | 34 ++++++++++++++++++----------------
 3 files changed, 39 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
index 56e85ab166c7..66dfc3c164e6 100644
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 struct vm_uffd_ops {
 	/* Checks if a VMA can support userfaultfd */
 	bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+	/*
+	 * Called to resolve UFFDIO_CONTINUE request.
+	 * Should return the folio found at pgoff in the VMA's pagecache if it
+	 * exists or ERR_PTR otherwise.
+	 * The returned folio is locked and with reference held.
+	 */
+	struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
 };
 
 /* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
index f2a25805b9bf..7bd887b64f62 100644
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
 	return ret;
 }
 
+static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+	struct folio *folio;
+	int err;
+
+	err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
+	if (err)
+		return ERR_PTR(err);
+
+	return folio;
+}
+
 static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
 {
 	return true;
 }
 
 static const struct vm_uffd_ops shmem_uffd_ops = {
-	.can_userfault	= shmem_can_userfault,
+	.can_userfault		= shmem_can_userfault,
+	.get_folio_noalloc	= shmem_get_folio_noalloc,
 };
 #endif /* CONFIG_USERFAULTFD */
 
mm/userfaultfd.c +18 -16
index e3024a39c19d..832dbdde5868 100644
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)
 	struct userfaultfd_ctx *ctx = state->ctx;
 	uffd_flags_t flags = state->flags;
 	struct vm_area_struct *dst_vma;
+	const struct vm_uffd_ops *ops;
 	int err;
 
 	/*
@@ -232,10 +233,12 @@ static int mfill_get_vma(struct mfill_state *state)
 	if (is_vm_hugetlb_page(dst_vma))
 		return 0;
 
-	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
+	ops = vma_uffd_ops(dst_vma);
+	if (!ops)
 		goto out_unlock;
-	if (!vma_is_shmem(dst_vma) &&
-	    uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
+
+	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE) &&
+	    !ops->get_folio_noalloc)
 		goto out_unlock;
 
 	return 0;
@@ -575,6 +578,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 static int mfill_atomic_pte_continue(struct mfill_state *state)
 {
 	struct vm_area_struct *dst_vma = state->vma;
+	const struct vm_uffd_ops *ops = vma_uffd_ops(dst_vma);
 	unsigned long dst_addr = state->dst_addr;
 	pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
 	struct inode *inode = file_inode(dst_vma->vm_file);
@@ -584,17 +588,16 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
 	struct page *page;
 	int ret;
 
-	ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
-	/* Our caller expects us to return -EFAULT if we failed to find folio */
-	if (ret == -ENOENT)
-		ret = -EFAULT;
-	if (ret)
-		goto out;
-	if (!folio) {
-		ret = -EFAULT;
-		goto out;
+	if (!ops) {
+		VM_WARN_ONCE(1, "UFFDIO_CONTINUE for unsupported VMA");
+		return -EOPNOTSUPP;
 	}
 
+	folio = ops->get_folio_noalloc(inode, pgoff);
+	/* Our caller expects us to return -EFAULT if we failed to find folio */
+	if (IS_ERR_OR_NULL(folio))
+		return -EFAULT;
+
 	page = folio_file_page(folio, pgoff);
 	if (PageHWPoison(page)) {
 		ret = -EIO;
@@ -607,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)
 		goto out_release;
 
 	folio_unlock(folio);
-	ret = 0;
-out:
-	return ret;
+	return 0;
+
 out_release:
 	folio_unlock(folio);
 	folio_put(folio);
-	goto out;
+	return ret;
 }
 
 /* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.53.0
author maintainer
#14 lore raw

PATCH v4 9/15 userfaultfd: introduce vm_uffd_ops->alloc_folio()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

---
 include/linux/userfaultfd_k.h |  6 +++
 mm/userfaultfd.c              | 92 ++++++++++++++++++-----------------
 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
index 66dfc3c164e6..55a67421de0a 100644
@@ -91,6 +91,12 @@ struct vm_uffd_ops {
 	 * The returned folio is locked and with reference held.
 	 */
 	struct folio *(*get_folio_noalloc)(struct inode *inode, pgoff_t pgoff);
+	/*
+	 * Called during resolution of UFFDIO_COPY request.
+	 * Should allocate and return a folio or NULL if allocation fails.
+	 */
+	struct folio *(*alloc_folio)(struct vm_area_struct *vma,
+				     unsigned long addr);
 };
 
 /* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44

142 lines not on this page · raw message

Signature · 1 line
2.53.0
author maintainer
#15 lore raw

PATCH v4 10/15 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.

---
 include/linux/shmem_fs.h      |  14 ----
 include/linux/userfaultfd_k.h |  19 +++--
 mm/shmem.c                    | 148 ++++++++++++----------------------
 mm/userfaultfd.c              |  80 +++++++++---------
 4 files changed, 106 insertions(+), 155 deletions(-)
include/linux/shmem_fs.h +0 -14
index a8273b32e041..1a345142af7d 100644
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)
 
 extern bool shmem_charge(struct inode *inode, long pages);
 
-#ifdef CONFIG_USERFAULTFD
-#ifdef CONFIG_SHMEM
-extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
-				  struct vm_area_struct *dst_vma,
-				  unsigned long dst_addr,
-				  unsigned long src_addr,
-				  uffd_flags_t flags,
-				  struct folio **foliop);
-#else /* !CONFIG_SHMEM */
-#define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \
-			       src_addr, flags, foliop) ({ BUG(); 0; })
-#endif /* CONFIG_SHMEM */
-#endif /* CONFIG_USERFAULTFD */
-
 /*
  * Used space is stored as unsigned 64-bit value in bytes but
  * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +14 -5
index 55a67421de0a..6f33307c2780 100644
@@ -97,6 +97,20 @@ struct vm_uffd_ops {
 	 */
 	struct folio *(*alloc_folio)(struct vm_area_struct *vma,
 				     unsigned long addr);
+	/*
+	 * Called during resolution of UFFDIO_COPY request.
+	 * Should only be called with a folio returned by alloc_folio() above.
+	 * The folio will be set to locked.
+	 * Returns 0 on success, error code on failure.
+	 */
+	int (*filemap_add)(struct folio *folio, struct vm_area_struct *vma,
+			 unsigned long addr);
+	/*
+	 * Called during resolution of UFFDIO_COPY request on the error
+	 * handling path.
+	 * Should revert the operation of ->filemap_add().
+	 */
+	void (*filemap_remove)(struct folio *folio, struct vm_area_struct *vma);
 };
 
 /* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
 /* Flags controlling behavior. These behavior changes are mode-independent. */
 #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
 
-extern int mfill_atomic_install_pte(pmd_t *dst_pmd,
-				    struct vm_area_struct *dst_vma,
-				    unsigned long dst_addr, struct page *page,
-				    bool newly_allocated, uffd_flags_t flags);
-
 extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 				 unsigned long src_start, unsigned long len,
 				 uffd_flags_t flags);
mm/shmem.c +53 -95

177 lines not on this page · raw message

mm/userfaultfd.c +39 -41

149 lines not on this page · raw message

Signature · 1 line
2.53.0
author maintainer
#16 lore raw

PATCH v4 11/15 userfaultfd: mfill_atomic(): remove retry logic

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().

Drop the retry logic from mfill_atomic().

---
 mm/userfaultfd.c | 24 ------------------------
 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
index e672a9e45d0c..935a3f6ebeed 100644
@@ -29,7 +29,6 @@ struct mfill_state {
 	struct vm_area_struct *vma;
 	unsigned long src_addr;
 	unsigned long dst_addr;
-	struct folio *folio;
 	pmd_t *pmd;
 };
 
@@ -899,7 +898,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(src_start + len <= src_start);
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
-retry:
 	err = mfill_get_vma(&state);
 	if (err)
 		goto out;
@@ -926,26 +924,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 		err = mfill_atomic_pte(&state);
 		cond_resched();
 
-		if (unlikely(err == -ENOENT)) {
-			void *kaddr;
-
-			mfill_put_vma(&state);
-			VM_WARN_ON_ONCE(!state.folio);
-
-			kaddr = kmap_local_folio(state.folio, 0);
-			err = copy_from_user(kaddr,
-					     (const void __user *)state.src_addr,
-					     PAGE_SIZE);
-			kunmap_local(kaddr);
-			if (unlikely(err)) {
-				err = -EFAULT;
-				goto out;
-			}
-			flush_dcache_folio(state.folio);
-			goto retry;
-		} else
-			VM_WARN_ON_ONCE(state.folio);
-
 		if (!err) {
 			state.dst_addr += PAGE_SIZE;
 			state.src_addr += PAGE_SIZE;
@@ -960,8 +938,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 
 	mfill_put_vma(&state);
 out:
-	if (state.folio)
-		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);
Signature · 1 line
2.53.0
author maintainer
#17 lore raw

Re: [PATCH v4 11/15] userfaultfd: mfill_atomic(): remove retry logic

Quoted 6 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().

Quoted 6 lines

Drop the retry logic from mfill_atomic().

---
 mm/userfaultfd.c | 24 ------------------------
 1 file changed, 24 deletions(-)

After discussion with David Carlier about potential replacement of VMA in mfill_copy_folio_retry(), I looked again in the code and realized that after all the rebases I didn't remove the bit that temporarily prevented returning ENOENT from __mfill_atomic_pte().

Andrew, can you please fold this into "userfaultfd: mfill_atomic(): remove retry logic"?

For a change it applies cleanly :)

commit 5173c8f4fd32f314907b3804217ef57d4e3a2220

Author: Mike Rapoport (Microsoft) <[email protected]>
Date:   Thu Apr 2 16:38:39 2026 +0300

userfaultfd: remove safety mesaure of not returning ENOENT from _copy

mm/userfaultfd.c +0 -3
index 935a3f6ebeed..dfd7094b1c40 100644
@@ -530,9 +530,6 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 		ops->filemap_remove(folio, state->vma);
 err_folio_put:
 	folio_put(folio);
-	/* Don't return -ENOENT so that our caller won't retry */
-	if (ret == -ENOENT)
-		ret = -EFAULT;
 	return ret;
 }
 
Signature · 2 lines
Sincerely yours,
Mike.
maintainer
#18 lore raw

Re: [PATCH v4 11/15] userfaultfd: mfill_atomic(): remove retry logic

Quoted 12 lines · On Thu, 2 Apr 2026 16:47:28 +0300 Mike Rapoport <[email protected]> wrote:
Quoted 6 lines

Drop the retry logic from mfill_atomic().

---
 mm/userfaultfd.c | 24 ------------------------
 1 file changed, 24 deletions(-)

After discussion with David Carlier about potential replacement of VMA in mfill_copy_folio_retry(), I looked again in the code and realized that after all the rebases I didn't remove the bit that temporarily prevented returning ENOENT from __mfill_atomic_pte().

Quoted 4 lines

Andrew, can you please fold this into "userfaultfd: mfill_atomic(): remove retry logic"?

For a change it applies cleanly :)

done.

Quoted 5 lines

commit 5173c8f4fd32f314907b3804217ef57d4e3a2220

Author: Mike Rapoport (Microsoft) <[email protected]>
Date:   Thu Apr 2 16:38:39 2026 +0300

userfaultfd: remove safety mesaure of not returning ENOENT from _copy

s/mesaure/measure/

Was "_copy" intended?

Quoted 2 lines

I instafolded this. End result:

From: "Mike Rapoport (Microsoft)" <[email protected]>
Subject: userfaultfd: mfill_atomic(): remove retry logic
Date: Thu, 2 Apr 2026 07:11:52 +0300

Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().

Drop the retry logic from mfill_atomic().

[[email protected]: remove safety measure of not returning ENOENT from _copy]
Cc Andrea Arcangeli <[email protected]>
Cc Andrei Vagin <[email protected]>
Cc Axel Rasmussen <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand (Arm) <[email protected]>
Cc Harry Yoo <[email protected]>
Cc Harry Yoo (Oracle) <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc James Houghton <[email protected]>
Cc Liam Howlett <[email protected]>
Cc Lorenzo Stoakes (Oracle) <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Muchun Song <[email protected]>
Cc Nikita Kalyazin <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Peter Xu <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Shuah Khan <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Vlastimil Babka <[email protected]>

---

 mm/userfaultfd.c |   27 ---------------------------
 1 file changed, 27 deletions(-)
mm/userfaultfd.c renamed from mm/userfaultfd.c~userfaultfd-mfill_atomic-remove-retry-logic +0 -27
@@ -29,7 +29,6 @@ struct mfill_state {
 	struct vm_area_struct *vma;
 	unsigned long src_addr;
 	unsigned long dst_addr;
-	struct folio *folio;
 	pmd_t *pmd;
 };
 
@@ -531,9 +530,6 @@ err_filemap_remove:
 		ops->filemap_remove(folio, state->vma);
 err_folio_put:
 	folio_put(folio);
-	/* Don't return -ENOENT so that our caller won't retry */
-	if (ret == -ENOENT)
-		ret = -EFAULT;
 	return ret;
 }
 
@@ -899,7 +895,6 @@ static __always_inline ssize_t mfill_ato
 	VM_WARN_ON_ONCE(src_start + len <= src_start);
 	VM_WARN_ON_ONCE(dst_start + len <= dst_start);
 
-retry:
 	err = mfill_get_vma(&state);
 	if (err)
 		goto out;
@@ -926,26 +921,6 @@ retry:
 		err = mfill_atomic_pte(&state);
 		cond_resched();
 
-		if (unlikely(err == -ENOENT)) {
-			void *kaddr;
-
-			mfill_put_vma(&state);
-			VM_WARN_ON_ONCE(!state.folio);
-
-			kaddr = kmap_local_folio(state.folio, 0);
-			err = copy_from_user(kaddr,
-					     (const void __user *)state.src_addr,
-					     PAGE_SIZE);
-			kunmap_local(kaddr);
-			if (unlikely(err)) {
-				err = -EFAULT;
-				goto out;
-			}
-			flush_dcache_folio(state.folio);
-			goto retry;
-		} else
-			VM_WARN_ON_ONCE(state.folio);
-
 		if (!err) {
 			state.dst_addr += PAGE_SIZE;
 			state.src_addr += PAGE_SIZE;
@@ -960,8 +935,6 @@ retry:
 
 	mfill_put_vma(&state);
 out:
-	if (state.folio)
-		folio_put(state.folio);
 	VM_WARN_ON_ONCE(copied < 0);
 	VM_WARN_ON_ONCE(err > 0);
 	VM_WARN_ON_ONCE(!copied && !err);

_

author maintainer
#19 lore raw

PATCH v4 12/15 mm: generalize handling of userfaults in __do_fault()

In reply to Mike Rapoport

From: Peter Xu <[email protected]>

When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:

- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist

Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().

For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.

Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.

---
 mm/memory.c      | 43 +++++++++++++++++++++++++++++++++++++++++++
 mm/shmem.c       | 12 ------------
 mm/userfaultfd.c |  9 +++++++++
 3 files changed, 52 insertions(+), 12 deletions(-)
mm/memory.c +43 -0
index 2f815a34d924..79c5328b26e3 100644
@@ -5329,6 +5329,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	return VM_FAULT_OOM;
 }
 
+#ifdef CONFIG_USERFAULTFD
+static vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct inode *inode;
+	struct folio *folio;
+
+	if (!(userfaultfd_missing(vma) || userfaultfd_minor(vma)))
+		return 0;
+
+	inode = file_inode(vma->vm_file);
+	folio = vma->vm_ops->uffd_ops->get_folio_noalloc(inode, vmf->pgoff);
+	if (!IS_ERR_OR_NULL(folio)) {
+		/*
+		 * TODO: provide a flag for get_folio_noalloc() to avoid
+		 * locking (or even the extra reference?)
+		 */
+		folio_unlock(folio);
+		folio_put(folio);
+		if (userfaultfd_minor(vma))
+			return handle_userfault(vmf, VM_UFFD_MINOR);
+	} else {
+		if (userfaultfd_missing(vma))
+			return handle_userfault(vmf, VM_UFFD_MISSING);
+	}
+
+	return 0;
+}
+#else
+static inline vm_fault_t __do_userfault(struct vm_fault *vmf)
+{
+	return 0;
+}
+#endif
+
 /*
  * The mmap_lock must have been held on entry, and may have been
  * released depending on flags and vma->vm_ops->fault() return value.
@@ -5361,6 +5396,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
 			return VM_FAULT_OOM;
 	}
 
+	/*
+	 * If this is a userfault trap, process it in advance before
+	 * triggering the genuine fault handler.
+	 */
+	ret = __do_userfault(vmf);
+	if (ret)
+		return ret;
+
 	ret = vma->vm_ops->fault(vmf);
 	if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
 			    VM_FAULT_DONE_COW)))
mm/shmem.c +0 -12
index 68620caaf75f..239545352cd2 100644
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	fault_mm = vma ? vma->vm_mm : NULL;
 
 	folio = filemap_get_entry(inode->i_mapping, index);
-	if (folio && vma && userfaultfd_minor(vma)) {
-		if (!xa_is_value(folio))
-			folio_put(folio);
-		*fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
-		return 0;
-	}
-
 	if (xa_is_value(folio)) {
 		error = shmem_swapin_folio(inode, index, &folio,
 					   sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	 * Fast cache lookup and swap lookup did not find it: allocate.
 	 */
 
-	if (vma && userfaultfd_missing(vma)) {
-		*fault_type = handle_userfault(vmf, VM_UFFD_MISSING);
-		return 0;
-	}
-
 	/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */
 	orders = shmem_allowable_huge_orders(inode, vma, index, write_end, false);
 	if (orders > 0) {
mm/userfaultfd.c +9 -0
index 935a3f6ebeed..9ba6ec8c0781 100644
@@ -2046,6 +2046,15 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 	    !vma_is_anonymous(vma))
 		return false;
 
+	/*
+	 * File backed VMAs (except HugeTLB) must implement
+	 * ops->get_folio_noalloc() because it's required by __do_userfault()
+	 * in page fault handling.
+	 */
+	if (!vma_is_anonymous(vma) && !is_vm_hugetlb_page(vma) &&
+	    !ops->get_folio_noalloc)
+		return false;
+
 	return ops->can_userfault(vma, vm_flags);
 }
 
Signature · 1 line
2.53.0
author maintainer
#20 lore raw

PATCH v4 13/15 KVM: guest_memfd: implement userfaultfd operations

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

---
 mm/filemap.c           |  1 +
 virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
index 406cef06b684..a91582293118 100644
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)
 
 	filemap_free_folio(mapping, folio);
 }
+EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
 
 /*
  * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +82 -2

114 lines not on this page · raw message

Signature · 1 line
2.53.0
#21 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 28 lines · Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

---
 mm/filemap.c           |  1 +
 virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +0 -0
index 406cef06b684..a91582293118 100644

mm/filemap.c void filemap_remove_folio(struct folio *folio) · line 265

Quoted 4 lines
mm/filemap.c +1 -0
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)
 
 	filemap_free_folio(mapping, folio);
 }
+EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");

This can be EXPORT_SYMBOL_FOR_KVM so that the symbol is exported if and only if KVM is built as a module.

Quoted 15 lines
mm/filemap.c +0 -0
 /*
  * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +1 -0
index 017d84a7adf3..46582feeed75 100644
@@ -7,6 +7,7 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include <linux/userfaultfd_k.h>
 
 #include "kvm_mm.h"
 

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 114

Quoted 7 lines
virt/kvm/guest_memfd.c +4 -0
@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
 }
 
+static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
+{
+	return __filemap_get_folio(inode->i_mapping, pgoff,
+				   FGP_LOCK | FGP_ACCESSED, 0);

Note, this will conflict with commit 6dad5447c7bf ("KVM: guest_memfd: Don't set FGP_ACCESSED when getting folios") sitting in

https://github.com/kvm-x86/linux.git gmem

I think the resolution is to just end up with:

static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)

{
	return filemap_lock_folio(inode->i_mapping, pgoff);
}

However, I think that'll be a moot point in the end (the conflict will be avoided). More below.

Quoted 20 lines
virt/kvm/guest_memfd.c +4 -2
+}
+
 /*
  * Returns a locked folio on success.  The caller is responsible for
  * setting the up-to-date flag before the memory is mapped into the guest.
@@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	 * Fast-path: See if folio is already present in mapping to avoid
 	 * policy_lookup.
 	 */
-	folio = __filemap_get_folio(inode->i_mapping, index,
-				    FGP_LOCK | FGP_ACCESSED, 0);
+	folio = kvm_gmem_get_folio_noalloc(inode, index);
 	if (!IS_ERR(folio))
 		return folio;
 
@@ -457,12 +463,86 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
 }
 #endif /* CONFIG_NUMA */
 
+#ifdef CONFIG_USERFAULTFD

virt/kvm/guest_memfd.c static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, · line 476

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	struct inode *inode = file_inode(vma->vm_file);
+
+	/*
+	 * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+	 * This ensures the memory can be mapped to userspace.
+	 */
+	if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+		return false;

I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.

Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.

My objection to an early, global "can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. for VM_UFFD_MISSING and VM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in __do_userfault(), not broadly when VM_UFFD_MINOR is added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path in kvm_gmem_fault_user_mapping().

I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the INIT_SHARED check to ->alloc_folio() and ->get_folio_noalloc(). And if we extract kvm_gmem_is_shared_mem() now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.

The bonus side effect is that dropping guest_memfd's more "complex" can_userfault means the only remaining check is constant based on the backing memory vs. the UFFD flags. If we want, the indirect call to a function can be replace with a constant vm_flags_t variable that enumerates the supported (or unsupported if we're feeling negative) flags, e.g.

include/linux/userfaultfd_k.h +2 -2
index 6f33307c2780..8a2d0625ffa3 100644
@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 
 /* VMA userfaultfd operations */
 struct vm_uffd_ops {
-       /* Checks if a VMA can support userfaultfd */
-       bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+       /* What UFFD flags/modes are supported. */
+       const vm_flags_t supported_uffd_flags;
        /*
         * Called to resolve UFFDIO_CONTINUE request.
         * Should return the folio found at pgoff in the VMA's pagecache if it

with usage like:

static const struct vm_uffd_ops shmem_uffd_ops = {
.supported_uffd_flags = __VM_UFFD_FLAGS,
.get_folio_noalloc = shmem_get_folio_noalloc,
.alloc_folio = shmem_mfill_folio_alloc,
.filemap_add = shmem_mfill_filemap_add,
.filemap_remove = shmem_mfill_filemap_remove,
};

[*] https://lore.kernel.org/all/[email protected]

Quoted 2 lines
virt/kvm/guest_memfd.c +2 -0
+	return true;
+}

...

virt/kvm/guest_memfd.c

Quoted 6 lines
virt/kvm/guest_memfd.c +6 -0
+static const struct vm_uffd_ops kvm_gmem_uffd_ops = {
+	.can_userfault     = kvm_gmem_can_userfault,
+	.get_folio_noalloc = kvm_gmem_get_folio_noalloc,
+	.alloc_folio       = kvm_gmem_folio_alloc,
+	.filemap_add       = kvm_gmem_filemap_add,
+	.filemap_remove    = kvm_gmem_filemap_remove,

Please use kvm_gmem_uffd_xxx(). The names are a bit verbose, but these are waaay to generic of names as-is, e.g. kvm_gmem_folio_alloc() has implications and restrictions far beyond just allocating a folio.

All in all, somelike like so (completely untested):

---
 include/linux/userfaultfd_k.h |  4 +-
 mm/filemap.c                  |  1 +
 mm/hugetlb.c                  |  8 +---
 mm/shmem.c                    |  7 +--
 mm/userfaultfd.c              |  6 +--
 virt/kvm/guest_memfd.c        | 80 ++++++++++++++++++++++++++++++++++-
 6 files changed, 87 insertions(+), 19 deletions(-)
include/linux/userfaultfd_k.h +2 -2
index 6f33307c2780..8a2d0625ffa3 100644
@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 
 /* VMA userfaultfd operations */
 struct vm_uffd_ops {
-	/* Checks if a VMA can support userfaultfd */
-	bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+	/* What UFFD flags/modes are supported. */
+	const vm_flags_t supported_uffd_flags;
 	/*
 	 * Called to resolve UFFDIO_CONTINUE request.
 	 * Should return the folio found at pgoff in the VMA's pagecache if it
mm/filemap.c +1 -0
index 6cd7974d4ada..19dfcebcd23f 100644
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)
 
 	filemap_free_folio(mapping, folio);
 }
+EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
 
 /*
  * page_cache_delete_batch - delete several folios from page cache
mm/hugetlb.c +1 -7
index 077968a8a69a..f55857961adb 100644
@@ -4819,14 +4819,8 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)
 }
 
 #ifdef CONFIG_USERFAULTFD
-static bool hugetlb_can_userfault(struct vm_area_struct *vma,
-				  vm_flags_t vm_flags)
-{
-	return true;
-}
-
 static const struct vm_uffd_ops hugetlb_uffd_ops = {
-	.can_userfault = hugetlb_can_userfault,
+	.supported_uffd_flags = __VM_UFFD_FLAGS,
 };
 #endif
 
mm/shmem.c +1 -6
index 239545352cd2..76d8488b9450 100644
@@ -3250,13 +3250,8 @@ static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
 	return folio;
 }
 
-static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
-{
-	return true;
-}
-
 static const struct vm_uffd_ops shmem_uffd_ops = {
-	.can_userfault		= shmem_can_userfault,
+	.supported_uffd_flags	= __VM_UFFD_FLAGS,
 	.get_folio_noalloc	= shmem_get_folio_noalloc,
 	.alloc_folio		= shmem_mfill_folio_alloc,
 	.filemap_add		= shmem_mfill_filemap_add,
mm/userfaultfd.c +3 -3
index 9ba6ec8c0781..ccbd7bb334c2 100644
@@ -58,8 +58,8 @@ static struct folio *anon_alloc_folio(struct vm_area_struct *vma,
 }
 
 static const struct vm_uffd_ops anon_uffd_ops = {
-	.can_userfault	= anon_can_userfault,
-	.alloc_folio	= anon_alloc_folio,
+	.supported_uffd_flags	= __VM_UFFD_FLAGS & ~VM_UFFD_MINOR,
+	.alloc_folio		= anon_alloc_folio,
 };
 
 static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma)
@@ -2055,7 +2055,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 	    !ops->get_folio_noalloc)
 		return false;
 
-	return ops->can_userfault(vma, vm_flags);
+	return ops->supported_uffd_flags & vm_flags;
 }
 
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
virt/kvm/guest_memfd.c +79 -1

110 lines not on this page · raw message

base-commit: d63beb006dba56d5fa219f106c7a97eb128c356f --

#22 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 112 lines · Sean Christopherson
Quoted 27 lines · Mike Rapoport

From: Nikita Kalyazin <[email protected]>

userfaultfd notifications about page faults used for live migration and snapshotting of VMs.

MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in
commit 7677f7fd8be7 ("userfaultfd: add minor fault registration
mode").

To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.

Add implementation of vm_uffd_ops to guest_memfd.

by: Mike Rapoport (Microsoft) <[email protected]> Signed-off-by:
Mike Rapoport (Microsoft) <[email protected]> --- mm/
filemap.c | 1 + virt/kvm/guest_memfd.c | 84 ++++++++++++
+++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)

mm/filemap.c index renamed from mm/filemap.c +0 -0

406cef06b684..a91582293118 100644 --- a/mm/filemap.c +++ b/mm/ filemap.c @@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)

Quoted 2 lines

filemap_free_folio(mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");

This can be EXPORT_SYMBOL_FOR_KVM so that the symbol is exported if and only if KVM is built as a module.

Quoted 14 lines

/* * page_cache_delete_batch - delete several folios from page
cache diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 017d84a7adf3..46582feeed75 100644 --- a/virt/kvm/ guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -7,6 +7,7 @@ #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h>

#include "kvm_mm.h"

mm/filemap.c index renamed from mm/filemap.c +1 -0
@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm
 *kvm, struct kvm_memory_slot *slot, return
 __kvm_gmem_prepare_folio(kvm, slot, index, folio); }
 
+static struct folio *kvm_gmem_get_folio_noalloc(struct inode
 *inode, pgoff_t pgoff) +{ +     return __filemap_get_folio(inode-
Quoted 1 line

i_mapping, pgoff, + FGP_LOCK |

Quoted 1 line

FGP_ACCESSED, 0);

Note, this will conflict with commit 6dad5447c7bf ("KVM:
guest_memfd: Don't set FGP_ACCESSED when getting folios") sitting in

https://github.com/kvm-x86/linux.git gmem

I think the resolution is to just end up with:

static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) { return filemap_lock_folio(inode->i_mapping, pgoff); }

However, I think that'll be a moot point in the end (the conflict will be avoided). More below.

Quoted 14 lines

+} + /* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest. @@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) * Fast-
path: See if folio is already present in mapping to avoid *
policy_lookup. */ - folio = __filemap_get_folio(inode-

Quoted 1 line

i_mapping, index, - FGP_LOCK |

FGP_ACCESSED, 0); + folio = kvm_gmem_get_folio_noalloc(inode,
index); if (!IS_ERR(folio)) return folio;

mm/filemap.c index renamed from mm/filemap.c +0 -0
@@ -457,12 +463,86 @@ static struct mempolicy
 *kvm_gmem_get_policy(struct vm_area_struct *vma, } #endif /*

CONFIG_NUMA */

Quoted 7 lines

+#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t
vm_flags) +{ + struct inode *inode = file_inode(vma->vm_file);
+ + /* + * Only support userfaultfd for guest_memfd with
INIT_SHARED flag. + * This ensures the memory can be mapped
to userspace. + */ + if (!(GMEM_I(inode)->flags &
GUEST_MEMFD_FLAG_INIT_SHARED)) + return false;

I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.

Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.

My objection to an early, global "can_userfault()" check is that
it's guaranteed to cause TOCTOU issues. E.g. for VM_UFFD_MISSING
and VM_UFFD_MINOR, the check on whether or not a given address can
be faulted in needs to happen in __do_userfault(), not broadly when
VM_UFFD_MINOR is added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path in kvm_gmem_fault_user_mapping().

I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the INIT_SHARED check to ->alloc_folio() and ->get_folio_noalloc(). And if we extract kvm_gmem_is_shared_mem() now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.

Quoted 5 lines

The bonus side effect is that dropping guest_memfd's more "complex" can_userfault means the only remaining check is constant based on the backing memory vs. the UFFD flags. If we want, the indirect
call to a function can be replace with a constant vm_flags_t
variable that enumerates the supported (or unsupported if we're
feeling negative) flags, e.g.

Thanks Sean. Checking for GUEST_MEMFD_FLAG_INIT_SHARED at the time of use and adding uffd_ prefixes to the callbacks make sense to me. I tested your changes in my local setup and they are functional with minor tweaks:
- remove the no-longer-used anon_can_userfault
- fix for the vm_flags check

mm/userfaultfd.c +1 -9
index 100aeadd7180..df91c40c6281 100644
@@ -32,14 +32,6 @@ struct mfill_state {
 	pmd_t *pmd;
 };
 
-static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
-{
-	/* anonymous memory does not support MINOR mode */
-	if (vm_flags & VM_UFFD_MINOR)
-		return false;
-	return true;
-}
-
 static struct folio *anon_alloc_folio(struct vm_area_struct *vma,
 				      unsigned long addr)
 {
@@ -2051,7 +2043,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 	    !ops->get_folio_noalloc)
 		return false;
 
-	return ops->supported_uffd_flags & vm_flags;
+	return (ops->supported_uffd_flags & vm_flags) == vm_flags;
 }
 
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
Quoted 183 lines
include/linux/ renamed from include/linux/userfaultfd_k.h +0 -0

userfaultfd_k.h index 6f33307c2780..8a2d0625ffa3 100644 --- a/ include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);

/* VMA userfaultfd operations */ struct vm_uffd_ops { - /*
Checks if a VMA can support userfaultfd */ - bool
(*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+ /* What UFFD flags/modes are supported. */ + const
vm_flags_t supported_uffd_flags; /* * Called to resolve UFFDIO_CONTINUE request. * Should return the folio found at pgoff in the VMA's pagecache if it

with usage like:

static const struct vm_uffd_ops shmem_uffd_ops = { .supported_uffd_flags = __VM_UFFD_FLAGS, .get_folio_noalloc =

shmem_get_folio_noalloc, .alloc_folio            =
shmem_mfill_folio_alloc, .filemap_add            =
shmem_mfill_filemap_add, .filemap_remove         =

shmem_mfill_filemap_remove, };

[*] https://lore.kernel.org/all/[email protected]

Quoted 1 line

+ return true; +}

...

mm/filemap.c index

Quoted 6 lines
mm/filemap.c index renamed from mm/filemap.c +6 -0
+static const struct vm_uffd_ops kvm_gmem_uffd_ops = { 
+     .can_userfault     = kvm_gmem_can_userfault, 
+     .get_folio_noalloc = kvm_gmem_get_folio_noalloc, 
+     .alloc_folio       = kvm_gmem_folio_alloc, 
+     .filemap_add       = kvm_gmem_filemap_add, 
+     .filemap_remove    = kvm_gmem_filemap_remove,

Please use kvm_gmem_uffd_xxx(). The names are a bit verbose, but these are waaay to generic of names as-is, e.g. kvm_gmem_folio_alloc() has implications and restrictions far beyond just allocating a folio.

All in all, somelike like so (completely untested):

--- include/linux/userfaultfd_k.h | 4 +- mm/
filemap.c | 1 + mm/hugetlb.c | 8

+--- mm/shmem.c                    |  7 +-- mm/
userfaultfd.c              |  6 +-- virt/kvm/guest_memfd.c        |

80 ++++++++++++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 19 deletions(-)

include/linux/ renamed from include/linux/userfaultfd_k.h +0 -0

userfaultfd_k.h index 6f33307c2780..8a2d0625ffa3 100644 --- a/ include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);

/* VMA userfaultfd operations */ struct vm_uffd_ops { - /*
Checks if a VMA can support userfaultfd */ - bool
(*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+ /* What UFFD flags/modes are supported. */ + const
vm_flags_t supported_uffd_flags; /* * Called to resolve
UFFDIO_CONTINUE request. * Should return the folio found at pgoff in
the VMA's pagecache if it diff --git a/mm/filemap.c b/mm/filemap.c index 6cd7974d4ada..19dfcebcd23f 100644 --- a/mm/filemap.c +++ b/mm/ filemap.c @@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio
*folio)

filemap_free_folio(mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");

/* * page_cache_delete_batch - delete several folios from page cache diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 077968a8a69a..f55857961adb 100644 --- a/mm/hugetlb.c +++ b/mm/
hugetlb.c @@ -4819,14 +4819,8 @@ static vm_fault_t
hugetlb_vm_op_fault(struct vm_fault *vmf) }

#ifdef CONFIG_USERFAULTFD -static bool hugetlb_can_userfault(struct
vm_area_struct *vma, - vm_flags_t
vm_flags) -{ - return true; -} - static const struct
vm_uffd_ops hugetlb_uffd_ops = { - .can_userfault =
hugetlb_can_userfault, + .supported_uffd_flags =
__VM_UFFD_FLAGS, }; #endif

mm/shmem.c index renamed from mm/shmem.c +0 -0

239545352cd2..76d8488b9450 100644 --- a/mm/shmem.c +++ b/mm/shmem.c

mm/shmem.c index renamed from mm/shmem.c +0 -1
@@ -3250,13 +3250,8 @@ static struct folio
 *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) return
 folio; }
 
-static bool shmem_can_userfault(struct vm_area_struct *vma,

vm_flags_t vm_flags) -{ - return true; -} - static const
struct vm_uffd_ops shmem_uffd_ops = {

mm/shmem.c index renamed from mm/shmem.c +1 -1
-       .can_userfault          = shmem_can_userfault, 
+       .supported_uffd_flags   = __VM_UFFD_FLAGS, 

.get_folio_noalloc = shmem_get_folio_noalloc, .alloc_folio = shmem_mfill_folio_alloc, .filemap_add = shmem_mfill_filemap_add, diff --git a/mm/
userfaultfd.c b/mm/userfaultfd.c index 9ba6ec8c0781..ccbd7bb334c2 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -58,8 +58,8 @@ static struct folio *anon_alloc_folio(struct vm_area_struct *vma, }

static const struct vm_uffd_ops anon_uffd_ops = {

mm/shmem.c index renamed from mm/shmem.c +2 -2
-       .can_userfault  = anon_can_userfault, 
-       .alloc_folio    = anon_alloc_folio, 
+       .supported_uffd_flags   = __VM_UFFD_FLAGS & ~VM_UFFD_MINOR, 
+       .alloc_folio            = anon_alloc_folio, };

static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma) @@ -2055,7 +2055,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !ops->get_folio_noalloc) return false;

- return ops->can_userfault(vma, vm_flags); + return ops-

Quoted 1 line

supported_uffd_flags & vm_flags; }

static void userfaultfd_set_vm_flags(struct vm_area_struct *vma, diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 462c5c5cb602..e634bf671d12 100644 --- a/virt/kvm/guest_memfd.c +++ b/
virt/kvm/guest_memfd.c @@ -7,6 +7,7 @@ #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h>

#include "kvm_mm.h"

mm/shmem.c index renamed from mm/shmem.c +0 -0
@@ -59,6 +60,11 @@ static pgoff_t kvm_gmem_get_index(struct

kvm_memory_slot *slot, gfn_t gfn) return gfn - slot->base_gfn + slot-

Quoted 1 line

gmem.pgoff; }

+static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t
index) +{ + return GMEM_I(inode)->flags &
GUEST_MEMFD_FLAG_INIT_SHARED; +} + static int
__kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot
*slot, pgoff_t index, struct folio *folio) { @@ -396,7 +402,7 @@
static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS;

mm/shmem.c index renamed from mm/shmem.c +1 -1
-       if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)) 
+       if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) return

VM_FAULT_SIGBUS;

folio = kvm_gmem_get_folio(inode, vmf->pgoff); @@ -456,12 +462,84 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, } #endif /* CONFIG_NUMA */

+#ifdef CONFIG_USERFAULTFD +static struct folio *kvm_gmem_uffd_get_folio_noalloc(struct inode *inode,

mm/shmem.c index renamed from mm/shmem.c +3 -0
+                                                    pgoff_t pgoff) 
+{ +       if (!kvm_gmem_is_shared_mem(inode, pgoff)) 
+               return NULL; + +       return

filemap_lock_folio(inode->i_mapping, pgoff); +} + +static struct folio *kvm_gmem_uffd_folio_alloc(struct vm_area_struct *vma,

mm/shmem.c index renamed from mm/shmem.c +2 -0
+                                              unsigned long addr) 
+{ +       struct inode *inode = file_inode(vma->vm_file); +

pgoff_t pgoff = linear_page_index(vma, addr); + struct
mempolicy *mpol; + struct folio *folio; + gfp_t gfp; +

mm/shmem.c index renamed from mm/shmem.c +2 -0
+       if (unlikely(pgoff >= (i_size_read(inode) >> PAGE_SHIFT))) 
+               return NULL; + +       if (!

kvm_gmem_is_shared_mem(inode, pgoff)) + return NULL; +
+ gfp = mapping_gfp_mask(inode->i_mapping); + mpol =
mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff); +
mpol = mpol ?: get_task_policy(current); + folio =
filemap_alloc_folio(gfp, 0, mpol); + mpol_cond_put(mpol); +
+ return folio; +} + +static int
kvm_gmem_uffd_filemap_add(struct folio *folio,

mm/shmem.c index renamed from mm/shmem.c +2 -0
+                                    struct vm_area_struct *vma, 
+                                    unsigned long addr) +{ +

struct inode *inode = file_inode(vma->vm_file); + struct
address_space *mapping = inode->i_mapping; + pgoff_t pgoff =
linear_page_index(vma, addr); + int err; + +
__folio_set_locked(folio); + err = filemap_add_folio(mapping,
folio, pgoff, GFP_KERNEL); + if (err) { +
folio_unlock(folio); + return err; + } + +
return 0; +} + +static void kvm_gmem_uffd_filemap_remove(struct
folio *folio, + struct
vm_area_struct *vma) +{ + filemap_remove_folio(folio); +
folio_unlock(folio); +} + +static const struct vm_uffd_ops
kvm_gmem_uffd_ops = { + .supported_uffd_flags =
__VM_UFFD_FLAGS, + .get_folio_noalloc =
kvm_gmem_uffd_get_folio_noalloc, + .alloc_folio =

kvm_gmem_uffd_folio_alloc, +       .filemap_add            =
kvm_gmem_uffd_filemap_add, +       .filemap_remove         =

kvm_gmem_uffd_filemap_remove, +}; +#endif /* CONFIG_USERFAULTFD */ + static const struct vm_operations_struct kvm_gmem_vm_ops = { .fault = kvm_gmem_fault_user_mapping, #ifdef CONFIG_NUMA .get_policy = kvm_gmem_get_policy, .set_policy =
kvm_gmem_set_policy, #endif +#ifdef CONFIG_USERFAULTFD
+ .uffd_ops = &kvm_gmem_uffd_ops, +#endif };

static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)

base-commit: d63beb006dba56d5fa219f106c7a97eb128c356f --

author maintainer
#23 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

In reply to Sean Christopherson

Hi Sean,

Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Mike Rapoport
(quoted diff) +10 -0
+#ifdef CONFIG_USERFAULTFD
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	struct inode *inode = file_inode(vma->vm_file);
+
+	/*
+	 * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+	 * This ensures the memory can be mapped to userspace.
+	 */
+	if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
Quoted 8 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+		return false;

I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.

I'm not against solving it as a part of uffd support, but since we are very close to the merge window, for now I asked Andrew to drop drop guest_memfd patches from the set and only move forward with the refactoring of uffd and shmem that has value on its own.

Quoted 14 lines

Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.

My objection to an early, global "can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. for VM_UFFD_MISSING and VM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in __do_userfault(), not broadly when VM_UFFD_MINOR is added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path in kvm_gmem_fault_user_mapping().

include/linux/userfaultfd_k.h +0 -0
index 6f33307c2780..8a2d0625ffa3 100644

include/linux/userfaultfd_k.h extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason); · line 86

Quoted 7 lines
include/linux/userfaultfd_k.h +2 -2
@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);
 
 /* VMA userfaultfd operations */
 struct vm_uffd_ops {
-       /* Checks if a VMA can support userfaultfd */
-       bool (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags);
+       /* What UFFD flags/modes are supported. */
+       const vm_flags_t supported_uffd_flags;

VMA maintainers really didn't like a fields flag in vm_uffd_ops when it was proposed earlier, but an indirect call may work.

Quoted 16 lines
include/linux/userfaultfd_k.h +0 -0
        /*
         * Called to resolve UFFDIO_CONTINUE request.
         * Should return the folio found at pgoff in the VMA's pagecache if it

with usage like:

static const struct vm_uffd_ops shmem_uffd_ops = {
.supported_uffd_flags = __VM_UFFD_FLAGS,
.get_folio_noalloc = shmem_get_folio_noalloc,
.alloc_folio = shmem_mfill_folio_alloc,
.filemap_add = shmem_mfill_filemap_add,
.filemap_remove = shmem_mfill_filemap_remove,
};

All in all, somelike like so (completely untested):

Quoted 8 lines
---
 include/linux/userfaultfd_k.h |  4 +-
 mm/filemap.c                  |  1 +
 mm/hugetlb.c                  |  8 +---
 mm/shmem.c                    |  7 +--
 mm/userfaultfd.c              |  6 +--
 virt/kvm/guest_memfd.c        | 80 ++++++++++++++++++++++++++++++++++-
 6 files changed, 87 insertions(+), 19 deletions(-)

Let's revisit after -rc1 :)

Signature · 2 lines
Sincerely yours,
Mike.
#24 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

In reply to Sean Christopherson
Quoted 12 lines · Sean Christopherson

[...snip...]

Review

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_USERFAULTFD
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	struct inode *inode = file_inode(vma->vm_file);
+
+	/*
+	 * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+	 * This ensures the memory can be mapped to userspace.
+	 */

Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?

Quoted 10 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+		return false;

I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.

Thanks :)

Quoted 9 lines

Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.

My objection to an early, global "can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. for VM_UFFD_MISSING and VM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in __do_userfault(), not broadly when VM_UFFD_MINOR is added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path in kvm_gmem_fault_user_mapping().

Quoted 7 lines

I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the INIT_SHARED check to ->alloc_folio() and ->get_folio_noalloc(). And if we extract kvm_gmem_is_shared_mem() now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.

I think it would help if we do some refactoring of existing allocation functions to provide guest_memfd's uffd ->alloc_folio and ->get_folio_noalloc callbacks instead of having new functions.

Perhaps uffd ->alloc_folio and ->get_folio_noalloc can call an alloc function refactored out of kvm_gmem_get_folio(), and also do the kvm_gmem_is_shared_mem() check.

[email protected], who is starting work on guest_memfd preservation (as in LUO/KHO).

Quoted 3 lines

[...snip...]

#25 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

[...snip...]

Quoted 12 lines
Quoted 9 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_USERFAULTFD
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	struct inode *inode = file_inode(vma->vm_file);
+
+	/*
+	 * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+	 * This ensures the memory can be mapped to userspace.
+	 */

Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?

Yes? It's more the opposite though: memory that can't be SHARED and thus can't be mapped into userspace can't possibly support userfaultfd. Six of one, half dozen of the other, but I think it's important to highlight that there's no judgment call being made, i.e. we aren't deciding to support only SHARED just because we're lazy, it's a hard requirement.

#26 lore raw

Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

[...snip...]

Quoted 12 lines
Quoted 9 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_USERFAULTFD
+static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)
+{
+	struct inode *inode = file_inode(vma->vm_file);
+
+	/*
+	 * Only support userfaultfd for guest_memfd with INIT_SHARED flag.
+	 * This ensures the memory can be mapped to userspace.
+	 */

Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?

Quoted 5 lines

Yes? It's more the opposite though: memory that can't be SHARED and thus can't be mapped into userspace can't possibly support userfaultfd. Six of one, half dozen of the other, but I think it's important to highlight that there's no judgment call being made, i.e. we aren't deciding to support only SHARED just because we're lazy, it's a hard requirement.

I guess this series is only intended to support some kinds of UFFD, but IIUC technically UFFDIO_CONTINUE can be used after the private memory is inserted (e.g. for TDX live migration).

author maintainer
#27 lore raw

PATCH v4 14/15 KVM: selftests: test userfaultfd minor for guest_memfd

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.

---
 .../testing/selftests/kvm/guest_memfd_test.c  | 113 ++++++++++++++++++
 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
index cc329b57ce2e..29f8d686c09f 100644
@@ -10,13 +10,17 @@
 #include <errno.h>
 #include <stdio.h>
 #include <fcntl.h>
+#include <pthread.h>
 
 #include <linux/bitmap.h>
 #include <linux/falloc.h>
 #include <linux/sizes.h>
+#include <linux/userfaultfd.h>
 #include <sys/mman.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/ioctl.h>
 
 #include "kvm_util.h"
 #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)
 	close(fd1);
 }
 
+struct fault_args {
+	char *addr;
+	char value;
+};
+
+static void *fault_thread_fn(void *arg)
+{
+	struct fault_args *args = arg;
+
+	/* Trigger page fault */
+	args->value = *args->addr;
+	return NULL;
+}
+
+static void test_uffd_minor(int fd, size_t total_size)
+{
+	struct uffdio_register uffd_reg;
+	struct uffdio_continue uffd_cont;
+	struct uffd_msg msg;
+	struct fault_args args;
+	pthread_t fault_thread;
+	void *mem, *mem_nofault, *buf = NULL;
+	int uffd, ret;
+	off_t offset = page_size;
+	void *fault_addr;
+	const char test_val = 0xcd;
+
+	ret = posix_memalign(&buf, page_size, total_size);
+	TEST_ASSERT_EQ(ret, 0);
+	memset(buf, test_val, total_size);
+
+	uffd = syscall(__NR_userfaultfd, O_CLOEXEC);
+	TEST_ASSERT(uffd != -1, "userfaultfd creation should succeed");
+
+	struct uffdio_api uffdio_api = {
+		.api = UFFD_API,
+		.features = 0,
+	};
+	ret = ioctl(uffd, UFFDIO_API, &uffdio_api);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_API) should succeed");
+
+	/* Map the guest_memfd twice: once with UFFD registered, once without */
+	mem = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem != MAP_FAILED, "mmap should succeed");
+
+	mem_nofault = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem_nofault != MAP_FAILED, "mmap should succeed");
+
+	/* Register UFFD_MINOR on the first mapping */
+	uffd_reg.range.start = (unsigned long)mem;
+	uffd_reg.range.len = total_size;
+	uffd_reg.mode = UFFDIO_REGISTER_MODE_MINOR;
+	ret = ioctl(uffd, UFFDIO_REGISTER, &uffd_reg);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_REGISTER) should succeed");
+
+	/*
+	 * Populate the page in the page cache first via mem_nofault.
+	 * This is required for UFFD_MINOR - the page must exist in the cache.
+	 * Write test data to the page.
+	 */
+	memcpy(mem_nofault + offset, buf + offset, page_size);
+
+	/*
+	 * Now access the same page via mem (which has UFFD_MINOR registered).
+	 * Since the page exists in the cache, this should trigger UFFD_MINOR.
+	 */
+	fault_addr = mem + offset;
+	args.addr = fault_addr;
+
+	ret = pthread_create(&fault_thread, NULL, fault_thread_fn, &args);
+	TEST_ASSERT(ret == 0, "pthread_create should succeed");
+
+	ret = read(uffd, &msg, sizeof(msg));
+	TEST_ASSERT(ret != -1, "read from userfaultfd should succeed");
+	TEST_ASSERT(msg.event == UFFD_EVENT_PAGEFAULT, "event type should be pagefault");
+	TEST_ASSERT((void *)(msg.arg.pagefault.address & ~(page_size - 1)) == fault_addr,
+		    "pagefault should occur at expected address");
+	TEST_ASSERT(msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR,
+		    "pagefault should be minor fault");
+
+	/* Resolve the minor fault with UFFDIO_CONTINUE */
+	uffd_cont.range.start = (unsigned long)fault_addr;
+	uffd_cont.range.len = page_size;
+	uffd_cont.mode = 0;
+	ret = ioctl(uffd, UFFDIO_CONTINUE, &uffd_cont);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_CONTINUE) should succeed");
+
+	/* Wait for the faulting thread to complete */
+	ret = pthread_join(fault_thread, NULL);
+	TEST_ASSERT(ret == 0, "pthread_join should succeed");
+
+	/* Verify the thread read the correct value */
+	TEST_ASSERT(args.value == test_val,
+		    "memory should contain the value that was written");
+	TEST_ASSERT(*(char *)(mem + offset) == test_val,
+		    "no further fault is expected");
+
+	ret = munmap(mem_nofault, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+
+	ret = munmap(mem, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+	free(buf);
+	close(uffd);
+}
+
 static void test_guest_memfd_flags(struct kvm_vm *vm)
 {
 	uint64_t valid_flags = vm_check_cap(vm, KVM_CAP_GUEST_MEMFD_FLAGS);
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)
 	gmem_test(file_size, vm, flags);
 	gmem_test(fallocate, vm, flags);
 	gmem_test(invalid_punch_hole, vm, flags);
+
+	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED)
+		gmem_test(uffd_minor, vm, flags);
 }
 
 static void test_guest_memfd(unsigned long vm_type)
Signature · 1 line
2.53.0
author maintainer
#28 lore raw

PATCH v4 15/15 KVM: selftests: test userfaultfd missing for guest_memfd

In reply to Mike Rapoport

From: Nikita Kalyazin <[email protected]>

The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.

---
 .../testing/selftests/kvm/guest_memfd_test.c  | 80 ++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
index 29f8d686c09f..eb29ee5d2991 100644
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)
 	close(uffd);
 }
 
+static void test_uffd_missing(int fd, size_t total_size)
+{
+	struct uffdio_register uffd_reg;
+	struct uffdio_copy uffd_copy;
+	struct uffd_msg msg;
+	struct fault_args args;
+	pthread_t fault_thread;
+	void *mem, *buf = NULL;
+	int uffd, ret;
+	off_t offset = page_size;
+	void *fault_addr;
+	const char test_val = 0xab;
+
+	ret = posix_memalign(&buf, page_size, total_size);
+	TEST_ASSERT_EQ(ret, 0);
+	memset(buf, test_val, total_size);
+
+	uffd = syscall(__NR_userfaultfd, O_CLOEXEC);
+	TEST_ASSERT(uffd != -1, "userfaultfd creation should succeed");
+
+	struct uffdio_api uffdio_api = {
+		.api = UFFD_API,
+		.features = 0,
+	};
+	ret = ioctl(uffd, UFFDIO_API, &uffdio_api);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_API) should succeed");
+
+	mem = mmap(NULL, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	TEST_ASSERT(mem != MAP_FAILED, "mmap should succeed");
+
+	uffd_reg.range.start = (unsigned long)mem;
+	uffd_reg.range.len = total_size;
+	uffd_reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+	ret = ioctl(uffd, UFFDIO_REGISTER, &uffd_reg);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_REGISTER) should succeed");
+
+	fault_addr = mem + offset;
+	args.addr = fault_addr;
+
+	ret = pthread_create(&fault_thread, NULL, fault_thread_fn, &args);
+	TEST_ASSERT(ret == 0, "pthread_create should succeed");
+
+	ret = read(uffd, &msg, sizeof(msg));
+	TEST_ASSERT(ret != -1, "read from userfaultfd should succeed");
+	TEST_ASSERT(msg.event == UFFD_EVENT_PAGEFAULT, "event type should be pagefault");
+	TEST_ASSERT((void *)(msg.arg.pagefault.address & ~(page_size - 1)) == fault_addr,
+		    "pagefault should occur at expected address");
+	TEST_ASSERT(!(msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP),
+		    "pagefault should not be write-protect");
+
+	uffd_copy.dst = (unsigned long)fault_addr;
+	uffd_copy.src = (unsigned long)(buf + offset);
+	uffd_copy.len = page_size;
+	uffd_copy.mode = 0;
+	ret = ioctl(uffd, UFFDIO_COPY, &uffd_copy);
+	TEST_ASSERT(ret != -1, "ioctl(UFFDIO_COPY) should succeed");
+
+	/* Wait for the faulting thread to complete - this provides the memory barrier */
+	ret = pthread_join(fault_thread, NULL);
+	TEST_ASSERT(ret == 0, "pthread_join should succeed");
+
+	/*
+	 * Now it's safe to check args.value - the thread has completed
+	 * and memory is synchronized
+	 */
+	TEST_ASSERT(args.value == test_val,
+		    "memory should contain the value that was copied");
+	TEST_ASSERT(*(char *)(mem + offset) == test_val,
+		    "no further fault is expected");
+
+	ret = munmap(mem, total_size);
+	TEST_ASSERT(!ret, "munmap should succeed");
+	free(buf);
+	close(uffd);
+}
+
 static void test_guest_memfd_flags(struct kvm_vm *vm)
 {
 	uint64_t valid_flags = vm_check_cap(vm, KVM_CAP_GUEST_MEMFD_FLAGS);
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)
 	gmem_test(fallocate, vm, flags);
 	gmem_test(invalid_punch_hole, vm, flags);
 
-	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED)
+	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED) {
 		gmem_test(uffd_minor, vm, flags);
+		gmem_test(uffd_missing, vm, flags);
+	}
 }
 
 static void test_guest_memfd(unsigned long vm_type)
Signature · 1 line
2.53.0
maintainer
#29 lore raw
In reply to Mike Rapoport
Quoted 1 line · On Thu, 2 Apr 2026 07:11:41 +0300 Mike Rapoport <[email protected]> wrote:

These patches enable support for userfaultfd in guest_memfd.

Thanks, I updated mm.git to this version.

Quoted 2 lines

v4 changes:
* fix lock leak not only in patch 4, but also in patch 2, thanks Harry

And.... this resulted in no overall alteration to mm.git, which broke my very surprised script!

iirc, this series was to fix a little bisection hole, so this is to be expected?