mm, kvm: allow uffd support in guest_memfd
Subsystems (10)
Mike Rapoport
maintainerv3 cover mm, kvm: allow uffd support in guest_memfd
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in guest_memfd.
As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.
In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.
Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
The handling of UFFDIO_MISSING in guest_memfd requires ability to remove a folio from page cache, the best way I could find was exporting filemap_remove_folio() to KVM.
I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.
Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.
v3 changes:
* add fixes from Harry and Andrei
* fix handling of WP-only mode for WP_ASYNC contexts in vma_can_userfault()
* address David's comments about mfill_get_pmd() and rename it to mfill_establish_pmd()
* add VM_WARN()s for unsupported operations (James)
* update comments using James' suggestions
v2: https://lore.kernel.org/all/[email protected]
* instead of returning uffd-specific values from ->fault() handlers add __do_userfault() helper to resolve user faults in __do_fault()
* address comments from Peter
* rebased on v7.0-c1
RFC: https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (11):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_establish_pmd() helper
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in
mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic(): remove retry logic
Nikita Kalyazin (3):
KVM: guest_memfd: implement userfaultfd operations
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd
Peter Xu (1):
mm: generalize handling of userfaults in __do_fault()
include/linux/mm.h | 5 + include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 73 +- mm/filemap.c | 1 + mm/hugetlb.c | 15 + mm/memory.c | 43 ++ mm/shmem.c | 188 ++--- mm/userfaultfd.c | 694 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 84 ++- 10 files changed, 860 insertions(+), 448 deletions(-)
base-commit: c369299895a591d96745d6492d4888259b004a9e --
2.53.0
Mike Rapoport
maintainerPATCH v3 1/15 userfaultfd: introduce mfill_copy_folio_locked() helper
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().
This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.
No functional change.
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/* +* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A +* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr); + +if (ret) +return -EFAULT ; + +flush_dcache_folio (folio); +return ret; +} + static int mfill_atomic_pte_copy(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,uffd_flags_t flags, struct folio **foliop) { - void *kaddr; int ret;struct folio *folio;
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,if (!folio) goto out; - kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (kaddr, (const void __user *) src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (kaddr); + ret =mfill_copy_folio_locked (folio, src_addr);/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* don't free the page */ goto out; } - -flush_dcache_folio (folio); }else { folio = *foliop; *foliop =NULL ;
Signature · 1 line
2.53.0
Re: [PATCH v3 01/15] userfaultfd: introduce mfill_copy_folio_locked() helper
Quoted 8 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from
mfill_atomic_pte_copy()into a helper functionmfill_copy_folio_locked().This makes improves code readability and makes complex
mfill_atomic_pte_copy()function easier to comprehend.
Quoted 6 lines
No functional change.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Acked-by Peter Xu <[email protected]>Reviewed-by David Hildenbrand (Arm) <[email protected]>---
Looks good to me,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerPATCH v3 2/15 userfaultfd: introduce struct mfill_state
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic() passes a lot of parameters down to its callees.
Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.
Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.
The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.
[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap() WARN]
--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)
mm/userfaultfd.c +82 -66
@@ -20,6 +20,20 @@#include "internal.h" #include "swap.h" +struct mfill_state { +struct userfaultfd_ctx *ctx; +unsigned long src_start; +unsigned long dst_start; +unsigned long len; +uffd_flags_t flags; + +struct vm_area_struct *vma; +unsigned long src_addr; +unsigned long dst_addr; +struct folio *folio; +pmd_t *pmd; +}; +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_atomic_pte_copy (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +unsigned long src_addr = state->src_addr; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio; +int ret; -if (!*foliop) { +if (!state->folio) { ret = -ENOMEM ; folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) { ret = -ENOENT ; - *foliop = folio; + state->folio = folio;/* don't free the page */ goto out; } }else { - folio = *foliop; - *foliop =NULL ; + folio = state->folio; + state->folio =NULL ; }/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,return ret; } -static int mfill_atomic_pte_zeropage (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_zeropage (struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl;int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, }/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ -static int mfill_atomic_pte_continue (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_continue (struct mfill_state *state) { -struct inode *inode =file_inode (dst_vma->vm_file); +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct inode *inode =file_inode (dst_vma->vm_file); +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio;struct page *page;int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ -static int mfill_atomic_pte_poison (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_poison (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma;struct mm_struct *dst_mm = dst_vma->vm_mm; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl; +int ret; _dst_pte =make_pte_marker (PTE_MARKER_POISONED ); ret = -EAGAIN ;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, uffd_flags_t flags);#endif /* CONFIG_HUGETLB_PAGE */ -static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long src_addr = state->src_addr; +unsigned long dst_addr = state->dst_addr; +struct folio **foliop = &state->folio; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;ssize_t err; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { -return mfill_atomic_pte_continue (dst_pmd, dst_vma, - dst_addr, flags); - }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) { -return mfill_atomic_pte_poison (dst_pmd, dst_vma, - dst_addr, flags); - } +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) +return mfill_atomic_pte_continue (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) +return mfill_atomic_pte_poison (state);/* * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,*/ if (!(dst_vma->vm_flags &VM_SHARED )) {if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); + err =mfill_atomic_pte_copy (state);else - err =mfill_atomic_pte_zeropage (dst_pmd, - dst_vma, dst_addr); + err =mfill_atomic_pte_zeropage (state); }else { err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long len, uffd_flags_t flags) { +struct mfill_state state = (struct mfill_state ){ + .ctx = ctx, + .dst_start = dst_start, + .src_start = src_start, + .flags = flags, + .len = len, + .src_addr = src_start, + .dst_addr = dst_start, + };struct mm_struct *dst_mm = ctx->mm;struct vm_area_struct *dst_vma; +long copied =0 ;ssize_t err;pmd_t *dst_pmd; -unsigned long src_addr, dst_addr; -long copied; -struct folio *folio;/* * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); - src_addr = src_start; - dst_addr = dst_start; - copied =0 ; - folio =NULL ; retry:/* * Make sure the vma is not shared, that the dst range is
@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; -while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma; -VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); + +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;
@@ -827,34 +843,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,* tables under us; pte_offset_map_lock() will deal with that. */ - err =mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, - src_addr, flags, &folio); + state.pmd = dst_pmd; + err =mfill_atomic_pte (&state);cond_resched ();if (unlikely (err == -ENOENT )) {void *kaddr;up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); -VM_WARN_ON_ONCE (!folio); +uffd_mfill_unlock (state.vma); +VM_WARN_ON_ONCE (!state.folio); - kaddr =kmap_local_folio (folio,0 ); + kaddr =kmap_local_folio (state.folio,0 ); err =copy_from_user (kaddr, - (const void __user *) src_addr, + (const void __user *)state.src_addr,PAGE_SIZE );kunmap_local (kaddr);if (unlikely (err)) { err = -EFAULT ; goto out; } -flush_dcache_folio (folio); +flush_dcache_folio (state.folio); goto retry; }else -VM_WARN_ON_ONCE (folio); +VM_WARN_ON_ONCE (state.folio);if (!err) { - dst_addr +=PAGE_SIZE ; - src_addr +=PAGE_SIZE ; + state.dst_addr +=PAGE_SIZE ; + state.src_addr +=PAGE_SIZE ; copied +=PAGE_SIZE ;if (fatal_signal_pending (current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.53.0
Re: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 26 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap()WARN]Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Harry Yoo <[email protected]>--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 807
Quoted 6 lines
mm/userfaultfd.c +1
-2@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; -while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma;
Oh wait, the lock leak was introduced in patch 2.
If there's an error between uffd_mfill_lock() and state.vma = dst_vma, it remains unlocked.
Probably should have been fixed in 2, not patch 4... Sorry didn't realize it earlier.
Quoted 12 lines
mm/userfaultfd.c +5
-2-VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); + +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 888
Quoted 10 lines
mm/userfaultfd.c +3
-3@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);
Sashiko raised a concern [2] that it the VMA might be unmapped and a new mapping created as a uffd hugetlb vma and leak the folio by going through
`if (is_vm_hugetlb_page(dst_vma))
return mfill_atomic_hugetlb(ctx, dst_vma, dst_start,
src_start, len, flags);`
but it appears to be a false positive (to me) because
if (atomic_read(&ctx->mmap_changing)) check should have detected unmapping and free the folio?
[2] https://sashiko.dev/#/patchset/20260330101116.1117699-1-rppt%40kernel.org?patch=13671
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 891
Quoted 3 lines
mm/userfaultfd.c +0
-0VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Otherwise looks correct to me.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerRe: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Hi Harry,
Quoted 30 lines · Harry Yoo (Oracle)
Quoted 29 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap()WARN]Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Harry Yoo <[email protected]>--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)mm/userfaultfd.c +0
-0@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock;
Quoted 5 lines
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 807
Quoted 3 lines
mm/userfaultfd.c +1
-2-while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma;Oh wait, the lock leak was introduced in patch 2.
Lock leak was introduced in patch 4 that moved getting the vma. Patch 2 missed the assignment of state.len and introduced an issue with bound checks.
Quoted 33 lines
If there's an error between
uffd_mfill_lock()andstate.vma = dst_vma, it remains unlocked.Probably should have been fixed in 2, not patch 4... Sorry didn't realize it earlier.
Quoted 12 lines
mm/userfaultfd.c +5
-2-VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); + +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 888
Quoted 10 lines
mm/userfaultfd.c +3
-3@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);Sashiko raised a concern [2] that it the VMA might be unmapped and a new mapping created as a uffd hugetlb vma and leak the folio by going through
Quoted 8 lines
`if (
is_vm_hugetlb_page(dst_vma))
return mfill_atomic_hugetlb(ctx,dst_vma,dst_start,
src_start, len, flags);`but it appears to be a false positive (to me) because
if (atomic_read(&ctx->mmap_changing))check should have detected unmapping and free the folio?
I think it's real, and it's there more or less from the beginning, although nobody hit it yet :)
Before retrying the copy we drop all the locks, so if the copy is really long the old mapping can be wiped and a new mapping can be created instead.
There's already a v4 of a patch that attempts to solve this:
https://lore.kernel.org/all/[email protected]
Quoted 11 lines
[2] https://sashiko.dev/#/patchset/20260330101116.1117699-1-rppt%40kernel.org?patch=13671
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 891
Quoted 3 lines
mm/userfaultfd.c +0
-0VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);Otherwise looks correct to me.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 33 lines · Mike Rapoport
Hi Harry,
Quoted 30 lines · Harry Yoo (Oracle)
Quoted 29 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap()WARN]Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Harry Yoo <[email protected]>--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)mm/userfaultfd.c +0
-0@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock;
Quoted 7 lines
Quoted 5 lines
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 807
Quoted 3 lines
mm/userfaultfd.c +1
-2-while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma;Oh wait, the lock leak was introduced in patch 2.
Lock leak was introduced in patch 4 that moved getting the vma.
Still not sure what I could possibly be missing, so let me try again. when I check out to this commit "userfaultfd: introduce struct mfill_state" I see:
| static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
| unsigned long dst_start,
| unsigned long src_start,
| unsigned long len,
| uffd_flags_t flags)
| {
| struct mfill_state state = (struct mfill_state){
| .ctx = ctx,
| .dst_start = dst_start,
| .src_start = src_start, .flags = flags,
| .len = len,
| .src_addr = src_start,
| .dst_addr = dst_start,
| };
|[ ...snip...]
| retry: | /* | * Make sure the vma is not shared, that the dst range is | * both valid and fully within a single existing vma. | */ | dst_vma = uffd_mfill_lock(dst_mm, dst_start, len);
It acquires the vma lock (or mmap_lock) here, but doesn't set state.vma.
| if (IS_ERR(dst_vma)) {
| err = PTR_ERR(dst_vma);
| goto out;
| }
|
| /*
| * If memory mappings are changing because of non-cooperative
| * operation (e.g. mremap) running in parallel, bail out and
| * request the user to retry later
| */
| down_read(&ctx->map_changing_lock);
| err = -EAGAIN;
| if (atomic_read(&ctx->mmap_changing))
| goto out_unlock;
|
| err = -EINVAL;
| /*
| * shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but
| * it will overwrite vm_ops, so vma_is_anonymous must return false.
| */
| if (WARN_ON_ONCE(vma_is_anonymous(dst_vma) &&
| dst_vma->vm_flags & VM_SHARED))
|
| /*
| * validate 'mode' now that we know the dst_vma: don't allow
| * a wrprotect copy if the userfaultfd didn't register as WP.
| */
| if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP))
| goto out_unlock;
|
| /*
| * If this is a HUGETLB vma, pass off to appropriate routine
| */
| if (is_vm_hugetlb_page(dst_vma))
| return mfill_atomic_hugetlb(ctx, dst_vma, dst_start,
| src_start, len, flags);
|
| if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
| goto out_unlock;
| if (!vma_is_shmem(dst_vma) &&
| uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE))
| goto out_unlock;
|
| state.vma = dst_vma;It is set here. So if anything before this jumps to out_unlock label due to a sanity check,
[...]
| while (state.src_addr < src_start + len) {
| VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
|
| pmd_t dst_pmdval;
| [...]
|
| out_unlock:
| up_read(&ctx->map_changing_lock);
| uffd_mfill_unlock(state.vma);the vma parameter will be NULL?
If I'm not missing something this is introduced in patch 2 and fixed in patch 4.
| out: | if (state.folio) | folio_put(state.folio); | VM_WARN_ON_ONCE(copied < 0); | VM_WARN_ON_ONCE(err > 0); | VM_WARN_ON_ONCE(!copied && !err); | return copied ? copied : err; | }
Quoted 22 lines
Quoted 22 lines
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 888
Quoted 11 lines
mm/userfaultfd.c +3
-3@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);Sashiko raised a concern [2] that it the VMA might be unmapped and a new mapping created as a uffd hugetlb vma and leak the folio by going through
`if (
is_vm_hugetlb_page(dst_vma))
return mfill_atomic_hugetlb(ctx,dst_vma,dst_start,
src_start, len, flags);`but it appears to be a false positive (to me) because
Quoted 8 lines
Quoted 2 lines
if (atomic_read(&ctx->mmap_changing))check should have detected unmapping and free the folio?I think it's real, and it's there more or less from the beginning, although nobody hit it yet :)
Before retrying the copy we drop all the locks, so if the copy is really long the old mapping can be wiped and a new mapping can be created instead.
Oops, perhaps I should have imagined harder :)
Quoted 3 lines
There's already a v4 of a patch that attempts to solve this:
Thanks for the pointer!
Quoted 5 lines
Quoted 5 lines
[2] https://sashiko.dev/#/patchset/20260330101116.1117699-1-rppt%40kernel.org?patch=13671
Quoted 3 lines
mm/userfaultfd.c +0
-0VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerRe: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 13 lines · Harry Yoo (Oracle)
On Tue, Mar 31, 2026 at 05:32:28PM +0300, Mike Rapoport wrote:
| /* | * Make sure the vma is not shared, that the dst range is | * both valid and fully within a single existing vma. | */ | dst_vma = uffd_mfill_lock(dst_mm, dst_start, len);It acquires the vma lock (or
mmap_lock) here, but doesn't set state.vma.| if (IS_ERR(dst_vma)) { | err = PTR_ERR(dst_vma); | goto out; | }
...
Quoted 23 lines
| if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma)) | goto out_unlock; | if (!vma_is_shmem(dst_vma) && | uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) | goto out_unlock; | | state.vma = dst_vma;It is set here. So if anything before this jumps to
out_unlocklabel due to a sanity check,[...]
| while (state.src_addr < src_start + len) { | VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len); | | pmd_t dst_pmdval; | [...] | | out_unlock: | up_read(&ctx->map_changing_lock); | uffd_mfill_unlock(state.vma);
Quoted 4 lines
the
vmaparameter will be NULL?If I'm not missing something this is introduced in patch 2 and fixed in patch 4.
You are right.
Here's a fixup (it causes a conflict in patch 4 though). Andrew, I can send v4 if you prefer.
mm/userfaultfd.c +1 -2
@@ -764,6 +764,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =PTR_ERR (dst_vma); goto out; } + state.vma = dst_vma;/* * If memory mappings are changing because of non-cooperative
@@ -804,8 +805,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; - state.vma = dst_vma; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);
Signature · 2 lines
Sincerely yours, Mike.
Andrew Morton
maintainerRe: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 2 lines · On Wed, 1 Apr 2026 10:36:03 +0300 Mike Rapoport <[email protected]> wrote:
Here's a fixup (it causes a conflict in patch 4 though). Andrew, I can send v4 if you prefer.
Looks pretty simple. "userfaultfd: introduce mfill_get_vma() and mfill_put_vma()" now leaves things like this:
retry:
err = mfill_get_vma(&state);
if (err)
goto out;
state.vma = dst_vma;
/*
* If this is a HUGETLB vma, pass off to appropriate routine
*/
if (is_vm_hugetlb_page(state.vma))
return mfill_atomic_hugetlb(ctx, state.vma, dst_start,
src_start, len, flags);
while (state.src_addr < src_start + len) {
VM_WARN_ON_ONCE(state.dst_addr >= dst_start + len);
err = mfill_establish_pmd(&state);
if (err)
break;
/*
* For shmem mappings, khugepaged is allowed to remove page
* tables under us; pte_offset_map_lock() will deal with that.
*/
err = mfill_atomic_pte(&state);
cond_resched();
if (unlikely(err == -ENOENT)) {
void *kaddr;
mfill_put_vma(&state);
VM_WARN_ON_ONCE(!state.folio);
kaddr = kmap_local_folio(state.folio, 0);
err = copy_from_user(kaddr,
(const void __user *)state.src_addr,
PAGE_SIZE);
kunmap_local(kaddr);
if (unlikely(err)) {
err = -EFAULT;
goto out;
}
flush_dcache_folio(state.folio);
goto retry;
} else
VM_WARN_ON_ONCE(state.folio);
if (!err) {
state.dst_addr += PAGE_SIZE;
state.src_addr += PAGE_SIZE;
copied += PAGE_SIZE;
if (fatal_signal_pending(current))
err = -EINTR;
}
if (err)
break;
}Andrew Morton
maintainerRe: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 7 lines · On Wed, 1 Apr 2026 10:37:17 -0700 Andrew Morton <[email protected]>…
Quoted 2 lines · On Wed, 1 Apr 2026 10:36:03 +0300 Mike Rapoport <[email protected]> wrote:
Here's a fixup (it causes a conflict in patch 4 though). Andrew, I can send v4 if you prefer.
Looks pretty simple. "userfaultfd: introduce
mfill_get_vma()andmfill_put_vma()" now leaves things like this:
err, actually, it gets messy.
Can we please have a v4? I think there are a few other things pending?
I've noted
https://lkml.kernel.org/r/[email protected] and
https://lkml.kernel.org/r/[email protected]
Alternatively, I can leave the current series in mm-unstable as-is and we can add a fixup at end-of-series. Not ideal, but it's OK.
Mike Rapoport
maintainerRe: [PATCH v3 02/15] userfaultfd: introduce struct mfill_state
Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Wed, 1 Apr 2026 10:37:17 -0700 Andrew Morton <[email protected]>…
Quoted 2 lines · On Wed, 1 Apr 2026 10:36:03 +0300 Mike Rapoport <[email protected]> wrote:
Here's a fixup (it causes a conflict in patch 4 though). Andrew, I can send v4 if you prefer.
Quoted 6 lines
Quoted 2 lines
Looks pretty simple. "userfaultfd: introduce
mfill_get_vma()andmfill_put_vma()" now leaves things like this:err, actually, it gets messy.
Can we please have a v4? I think there are a few other things pending?
Sent.
Quoted 2 lines
Alice fixed this on the Rust side and the fix is queued to char-misc-linus:
https://lore.kernel.org/all/CAH5fLgg63SYHQ34yqy7o2cA5f2Gh0aNTwTeQkzbWb+kV4Rr3Qw@mail.gmail.com
Quoted 1 line
We've been through this since Peter's first posting and I think there's a consensus about necessity of vm_uffd_ops and the argument was about what exact API they should have.
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v3 3/15 userfaultfd: introduce mfill_establish_pmd() helper
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.
Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().
--- mm/userfaultfd.c | 102 ++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 50 deletions(-)
mm/userfaultfd.c +52 -50
@@ -157,6 +157,56 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) +{ +pgd_t *pgd; +p4d_t *p4d; +pud_t *pud; + + pgd =pgd_offset (mm, address); + p4d =p4d_alloc (mm, pgd, address); +if (!p4d) +return NULL ; + pud =pud_alloc (mm, p4d, address); +if (!pud) +return NULL ; +/* +* Note that we didn't run this because the pmd was +* missing, the *pmd may be already established and in +* turn it may also be a trans_huge_pmd. +*/ +return pmd_alloc (mm, pud, address); +} + +static int mfill_establish_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd, dst_pmdval; + + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +if (unlikely (pmd_none (dst_pmdval)) && +unlikely (__pte_alloc (dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_leaf (dst_pmdval))) +return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +} +/* Check if dst_addr is outside of file's size. Must be called with ptl held. */ static bool mfill_file_over_size (struct vm_area_struct *dst_vma,unsigned long dst_addr)
@@ -489,27 +539,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)return ret; } -static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) -{ -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; - - pgd =pgd_offset (mm, address); - p4d =p4d_alloc (mm, pgd, address); -if (!p4d) -return NULL ; - pud =pud_alloc (mm, p4d, address); -if (!pud) -return NULL ; -/* -* Note that we didn't run this because the pmd was -* missing, the *pmd may be already established and in -* turn it may also be a trans_huge_pmd. -*/ -return pmd_alloc (mm, pud, address); -} -#ifdef CONFIG_HUGETLB_PAGE /* * mfill_atomic processing for HUGETLB vmas. Note that this routine is
@@ -742,7 +771,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err; -pmd_t *dst_pmd;/* * Sanitize the command parameters:
@@ -809,41 +837,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_establish_pmd (&state); +if (err)break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -if (unlikely (pmd_none (dst_pmdval)) && -unlikely (__pte_alloc (dst_mm, dst_pmd))) { - err = -ENOMEM ; -break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -/* -* If the dst_pmd is THP don't override it and just be strict. -* (This includes the case where the PMD used to be THP and -* changed back to none after __pte_alloc().) -*/ -if (unlikely (!pmd_present (dst_pmdval) || -pmd_trans_huge (dst_pmdval))) { - err = -EEXIST ; -break ; - } -if (unlikely (pmd_bad (dst_pmdval))) { - err = -EFAULT ; -break ; - }/* * For shmem mappings, khugepaged is allowed to remove page * tables under us; pte_offset_map_lock() will deal with that. */ - state.pmd = dst_pmd; err =mfill_atomic_pte (&state);cond_resched ();
Signature · 1 line
2.53.0
Re: [PATCH v3 03/15] userfaultfd: introduce mfill_establish_pmd() helper
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
There is a lengthy code chunk in
mfill_atomic()that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.
Quoted 5 lines
Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of
mfill_atomic_pte_copy().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>---
Looks good to me,
Quoted 18 lines
mm/userfaultfd.c | 102 ++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 50 deletions(-)mm/userfaultfd.c +11
-0+static int mfill_establish_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd, dst_pmdval; + + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); + if (unlikely(pmd_none(dst_pmdval)) &&
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 201
Quoted 10 lines
mm/userfaultfd.c +10
-0+ unlikely(__pte_alloc(dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_leaf (dst_pmdval)))
Just noting: ^ this was
pmd_trans_huge() but now it is pmd_leaf().
but since the present bit is separately checked and hugetlb vma is handled in a different path I don't expect functional change.
Quoted 7 lines
mm/userfaultfd.c +7
-0+return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +}
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerPATCH v3 4/15 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.
This function will be used later during refactoring of mfill_atomic_pte_copy().
Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.
--- mm/userfaultfd.c | 126 ++++++++++++++++++++++++++++------------------- 1 file changed, 75 insertions(+), 51 deletions(-)
mm/userfaultfd.c +75 -51
@@ -157,6 +157,75 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +if (!state->vma) +return ; + +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + state->vma = dst_vma; + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) + goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) +return 0 ; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock; + +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;
@@ -767,8 +836,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, .src_addr = src_start, .dst_addr = dst_start, }; -struct mm_struct *dst_mm = ctx->mm; -struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err;
@@ -783,57 +850,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (dst_start + len <= dst_start); retry: -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ - dst_vma =uffd_mfill_lock (dst_mm, dst_start, len); -if (IS_ERR (dst_vma)) { - err =PTR_ERR (dst_vma); + err =mfill_get_vma (&state); +if (err) goto out; - } - -/* -* If memory mappings are changing because of non-cooperative -* operation (e.g. mremap) running in parallel, bail out and -* request the user to retry later -*/ -down_read (&ctx->map_changing_lock); - err = -EAGAIN ; -if (atomic_read (&ctx->mmap_changing)) - goto out_unlock; - - err = -EINVAL ; -/* -* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but -* it will overwrite vm_ops, so vma_is_anonymous must return false. -*/ -if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && - dst_vma->vm_flags &VM_SHARED )) - goto out_unlock; - -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) - goto out_unlock;/* * If this is a HUGETLB vma, pass off to appropriate routine */ -if (is_vm_hugetlb_page (dst_vma)) -return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, +if (is_vm_hugetlb_page (state.vma)) +return mfill_atomic_hugetlb (ctx, state.vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; - - state.vma = dst_vma; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);
@@ -852,8 +879,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (unlikely (err == -ENOENT )) {void *kaddr; -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state);VM_WARN_ON_ONCE (!state.folio); kaddr =kmap_local_folio (state.folio,0 );
@@ -882,9 +908,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -out_unlock: -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 5/15 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).
In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.
Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.
As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.
--- mm/userfaultfd.c | 75 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 24 deletions(-)
mm/userfaultfd.c +51 -24
@@ -405,35 +405,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_establish_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) { +/* +* Fallback to copy_from_user outside mmap_lock. +* If retry is successful, mfill_copy_folio_locked() returns +* with locks retaken by mfill_get_vma(). +* If there was an error, we must mfill_put_vma() anyway and it +* will take care of unlocking if needed. +*/ + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto out_release; }/*
@@ -443,17 +471,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio); - ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;folio_put (folio); goto out; }
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 6/15 userfaultfd: move vma_can_userfault out of line
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault() has grown pretty big and it's not called on performance critical path.
Move it out of line.
No functional changes.
--- include/linux/userfaultfd_k.h | 35 ++--------------------------------- mm/userfaultfd.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags, -bool wp_async) -{ - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true; - -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && - !vma_is_anonymous (vma)) -return false; - -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async);static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {
mm/userfaultfd.c +33 -0
@@ -2018,6 +2018,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,return moved ? moved : err; } +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async) +{ + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + +if ((vm_flags &VM_UFFD_MINOR ) && + (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) +return false; + +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem & hugetlbfs are not supported but only +* anonymous. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && + !vma_is_anonymous (vma)) +return false; + +/* By default, allow any of anon|shmem|hugetlb */ +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +static void userfaultfd_set_vm_flags (struct vm_area_struct *vma,vm_flags_t vm_flags) {
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 7/15 userfaultfd: introduce vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core
MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.
Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.
Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.
Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.
--- include/linux/mm.h | 5 ++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 15 ++++++++++++ mm/shmem.c | 15 ++++++++++++ mm/userfaultfd.c | 44 ++++++++++++++++++++++++----------- 5 files changed, 72 insertions(+), 13 deletions(-)
include/linux/mm.h +5 -0
@@ -741,6 +741,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -826,6 +828,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;
mm/hugetlb.c +15 -0
@@ -4818,6 +4818,18 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops hugetlb_uffd_ops = { + .can_userfault = hugetlb_can_userfault, +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4831,6 +4843,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +15 -0
@@ -3294,6 +3294,15 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,shmem_inode_unacct_blocks (inode,1 ); return ret; } + +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops shmem_uffd_ops = { + .can_userfault = shmem_can_userfault, +};#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS
@@ -5313,6 +5322,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5322,6 +5334,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)
mm/userfaultfd.c +31 -13
71 lines not on this page · raw message
Signature · 1 line
2.53.0
Matthew Wilcox
maintainerRe: [PATCH v3 07/15] userfaultfd: introduce vm_uffd_ops
Quoted 11 lines · Mike Rapoport
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on
vma_is_anonymous(),vma_is_shmem()andis_vm_hugetlb_page()and some of these conditions can be expressed as operations implemented by a particular memory type.Introduce
vm_uffd_opsextension tovm_operations_structthat will delegate memory type specific operations to a VMA owner.
But the VMA owner already gets control of the page fault. We don't need the vm_uffd_ops to be accessible from the vm_operations_struct. The page faullt handler can pass a vm_uffd_ops into the uffd functions that it calls.
Mike Rapoport
maintainerRe: [PATCH v3 07/15] userfaultfd: introduce vm_uffd_ops
Quoted 10 lines · Matthew Wilcox
Quoted 9 lines · Mike Rapoport
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on
vma_is_anonymous(),vma_is_shmem()andis_vm_hugetlb_page()and some of these conditions can be expressed as operations implemented by a particular memory type.
Quoted 7 lines
Quoted 2 lines
Introduce
vm_uffd_opsextension tovm_operations_structthat will delegate memory type specific operations to a VMA owner.But the VMA owner already gets control of the page fault. We don't need the
vm_uffd_opsto be accessible from thevm_operations_struct. The page faullt handler can pass avm_uffd_opsinto the uffd functions that it calls.
Not sure I follow you here. The ops are not required by the page fault handler, they are needed for uffd ioctl handling, e.g
userfaultfd_ioctl(UFFDIO_COPY) -> mfill_atomic_copy() -> vma = find_vma(); /* do VMA specific stuff */
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v3 8/15 shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.
Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.
--- include/linux/userfaultfd_k.h | 7 +++++++ mm/shmem.c | 15 ++++++++++++++- mm/userfaultfd.c | 34 ++++++++++++++++++---------------- 3 files changed, 39 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* +* Called to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +* The returned folio is locked and with reference held. +*/ +struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); };/* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, return ret; } +static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +} +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) {return true; }static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .can_userfault = shmem_can_userfault, + .get_folio_noalloc = shmem_get_folio_noalloc, };#endif /* CONFIG_USERFAULTFD */
mm/userfaultfd.c +18 -16
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)struct userfaultfd_ctx *ctx = state->ctx;uffd_flags_t flags = state->flags;struct vm_area_struct *dst_vma; +const struct vm_uffd_ops *ops;int err;/*
@@ -232,10 +233,12 @@ static int mfill_get_vma(struct mfill_state *state)if (is_vm_hugetlb_page (dst_vma))return 0 ; -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + ops =vma_uffd_ops (dst_vma); +if (!ops) goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && + !ops->get_folio_noalloc) goto out_unlock;return 0 ;
@@ -575,6 +578,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)static int mfill_atomic_pte_continue (struct mfill_state *state) {struct vm_area_struct *dst_vma = state->vma; +const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);
@@ -584,17 +588,16 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); -/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; +if (!ops) { +VM_WARN_ONCE (1 ,"UFFDIO_CONTINUE for unsupported VMA" ); +return -EOPNOTSUPP ; } + folio = ops->get_folio_noalloc (inode, pgoff); +/* Our caller expects us to return -EFAULT if we failed to find folio */ +if (IS_ERR_OR_NULL (folio)) +return -EFAULT ; + page =folio_file_page (folio, pgoff);if (PageHWPoison (page)) { ret = -EIO ;
@@ -607,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 9/15 userfaultfd: introduce vm_uffd_ops->alloc_folio()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio
Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.
Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.
Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should allocate and return a folio or NULL if allocation fails. +*/ +struct folio *(*alloc_folio)(struct vm_area_struct *vma, +unsigned long addr); };/* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44
142 lines not on this page · raw message
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 10/15 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.
Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().
Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c
mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 19 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 80 +++++++++--------- 4 files changed, 106 insertions(+), 155 deletions(-)
include/linux/shmem_fs.h +0 -14
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +14 -5
@@ -97,6 +97,20 @@ struct vm_uffd_ops {*/ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should only be called with a folio returned by alloc_folio() above. +* The folio will be set to locked. +* Returns 0 on success, error code on failure. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);
mm/shmem.c +53 -95
177 lines not on this page · raw message
mm/userfaultfd.c +39 -41
149 lines not on this page · raw message
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 11/15 userfaultfd: mfill_atomic(): remove retry logic
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().
Drop the retry logic from mfill_atomic().
--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
@@ -29,7 +29,6 @@ struct mfill_state {struct vm_area_struct *vma;unsigned long src_addr;unsigned long dst_addr; -struct folio *folio;pmd_t *pmd; };
@@ -899,7 +898,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); -retry: err =mfill_get_vma (&state);if (err) goto out;
@@ -926,26 +924,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =mfill_atomic_pte (&state);cond_resched (); -if (unlikely (err == -ENOENT )) { -void *kaddr; - -mfill_put_vma (&state); -VM_WARN_ON_ONCE (!state.folio); - - kaddr =kmap_local_folio (state.folio,0 ); - err =copy_from_user (kaddr, - (const void __user *)state.src_addr, -PAGE_SIZE ); -kunmap_local (kaddr); -if (unlikely (err)) { - err = -EFAULT ; - goto out; - } -flush_dcache_folio (state.folio); - goto retry; - }else -VM_WARN_ON_ONCE (state.folio); -if (!err) { state.dst_addr +=PAGE_SIZE ; state.src_addr +=PAGE_SIZE ;
@@ -960,8 +938,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,mfill_put_vma (&state); out: -if (state.folio) -folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 12/15 mm: generalize handling of userfaults in __do_fault()
In reply to Mike RapoportFrom: Peter Xu <[email protected]>
When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:
- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist
Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().
For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.
Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.
--- mm/memory.c | 43 +++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 12 ------------ mm/userfaultfd.c | 9 +++++++++ 3 files changed, 52 insertions(+), 12 deletions(-)
mm/memory.c +43 -0
@@ -5329,6 +5329,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +#ifdef CONFIG_USERFAULTFD +static vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode; +struct folio *folio; + +if (!(userfaultfd_missing (vma) ||userfaultfd_minor (vma))) +return 0 ; + + inode =file_inode (vma->vm_file); + folio = vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff); +if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ +folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) +return handle_userfault (vmf,VM_UFFD_MINOR ); + }else { +if (userfaultfd_missing (vma)) +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +#else +static inline vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value.
@@ -5361,6 +5396,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/* +* If this is a userfault trap, process it in advance before +* triggering the genuine fault handler. +*/ + ret =__do_userfault (vmf); +if (ret) +return ret; + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY |VM_FAULT_DONE_COW )))
mm/shmem.c +0 -12
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {
mm/userfaultfd.c +9 -0
@@ -2046,6 +2046,15 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !vma_is_anonymous(vma))return false; +/* +* File backed VMAs (except HugeTLB) must implement +* ops->get_folio_noalloc() because it's required by __do_userfault() +* in page fault handling. +*/ +if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && + !ops->get_folio_noalloc) +return false; +return ops->can_userfault (vma, vm_flags); }
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 13/15 KVM: guest_memfd: implement userfaultfd operations
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").
To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.
Add implementation of vm_uffd_ops to guest_memfd.
--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +82 -2
114 lines not on this page · raw message
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 14/15 KVM: selftests: test userfaultfd minor for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
@@ -10,13 +10,17 @@#include <errno.h> #include <stdio.h> #include <fcntl.h> +#include <pthread.h> #include <linux/bitmap.h> #include <linux/falloc.h> #include <linux/sizes.h> +#include <linux/userfaultfd.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> #include "kvm_util.h" #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)close (fd1); } +struct fault_args { +char *addr; +char value; +}; + +static void *fault_thread_fn (void *arg) +{ +struct fault_args *args = arg; + +/* Trigger page fault */ + args->value = *args->addr; +return NULL ; +} + +static void test_uffd_minor (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_continue uffd_cont; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *mem_nofault, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xcd ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + +/* Map the guest_memfd twice: once with UFFD registered, once without */ + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + mem_nofault =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem_nofault !=MAP_FAILED ,"mmap should succeed" ); + +/* Register UFFD_MINOR on the first mapping */ + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MINOR ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + +/* +* Populate the page in the page cache first via mem_nofault. +* This is required for UFFD_MINOR - the page must exist in the cache. +* Write test data to the page. +*/ +memcpy (mem_nofault + offset, buf + offset, page_size); + +/* +* Now access the same page via mem (which has UFFD_MINOR registered). +* Since the page exists in the cache, this should trigger UFFD_MINOR. +*/ + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_MINOR , +"pagefault should be minor fault" ); + +/* Resolve the minor fault with UFFDIO_CONTINUE */ + uffd_cont.range.start = (unsigned long )fault_addr; + uffd_cont.range.len = page_size; + uffd_cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &uffd_cont); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_CONTINUE) should succeed" ); + +/* Wait for the faulting thread to complete */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* Verify the thread read the correct value */ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was written" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem_nofault, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (file_size, vm, flags);gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); + +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +gmem_test (uffd_minor, vm, flags); }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v3 15/15 KVM: selftests: test userfaultfd missing for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)close (uffd); } +static void test_uffd_missing (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_copy uffd_copy; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xab ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MISSING ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (!(msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ), +"pagefault should not be write-protect" ); + + uffd_copy.dst = (unsigned long )fault_addr; + uffd_copy.src = (unsigned long )(buf + offset); + uffd_copy.len = page_size; + uffd_copy.mode =0 ; + ret =ioctl (uffd,UFFDIO_COPY , &uffd_copy); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_COPY) should succeed" ); + +/* Wait for the faulting thread to complete - this provides the memory barrier */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* +* Now it's safe to check args.value - the thread has completed +* and memory is synchronized +*/ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was copied" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); -if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) {gmem_test (uffd_minor, vm, flags); +gmem_test (uffd_missing, vm, flags); + } }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 1 line · On Mon, 30 Mar 2026 13:11:01 +0300 Mike Rapoport <[email protected]> wrote:
These patches enable support for userfaultfd in
guest_memfd.
Thanks, I've updated mm.git's mm-unstable branch to this version. I added a little note-to-self to keep tabs on willy's [07/15] comment.
The series seems to be converging nicely. Several of the patches aren't showing R-b/A-b at this time.
I've moved this series even further down-queue, so it's definitely in the second-week-of-merge-window batch. So I'll be looking to move it into mm-stable around Monday of that week (Apr 27?). Four weeks for testing, review and little touchups.
Quoted 7 lines
v3 changes:
* add fixes from Harry and Andrei
* fix handling of WP-only mode forWP_ASYNCcontexts invma_can_userfault()
* address David's comments aboutmfill_get_pmd()and rename it tomfill_establish_pmd()
* add VM_WARN()s for unsupported operations (James)
* update comments using James' suggestions
Here's how v3 altered mm.git:
include/linux/userfaultfd_k.h | 6 +++--- mm/memory.c | 2 +- mm/userfaultfd.c | 12 ++++++++---- 3 files changed, 12 insertions(+), 8 deletions(-)
include/linux/userfaultfd_k.h renamed from include/linux/userfaultfd_k.h~b +3 -3
@@ -96,14 +96,14 @@ struct vm_uffd_ops {struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff);/* * Called during resolution of UFFDIO_COPY request. -* Should allocate and return a folio or NULL if allocation -* fails. +* Should allocate and return a folio or NULL if allocation fails. */ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr);/* * Called during resolution of UFFDIO_COPY request. -* Should lock the folio and add it to VMA's page cache. +* Should only be called with a folio returned by alloc_folio() above. +* The folio will be set to locked. * Returns 0 on success, error code on failure. */ int (*filemap_add)(struct folio *folio, struct vm_area_struct *vma,
mm/memory.c renamed from mm/memory.c~b +1 -1
@@ -5493,7 +5493,7 @@ static vm_fault_t __do_fault(struct vm_f }/* -* If this is an userfaultfd trap, process it in advance before +* If this is a userfault trap, process it in advance before * triggering the genuine fault handler. */ ret =__do_userfault (vmf);
mm/userfaultfd.c renamed from mm/userfaultfd.c~b +8 -4
@@ -502,7 +502,7 @@ static int __mfill_atomic_pte(struct mfi }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) {clear_user_highpage (&folio->page, state->dst_addr); }else { -VM_WARN_ONCE (1 ,"unknown UFFDIO operation" ); +VM_WARN_ONCE (1 ,"Unknown UFFDIO operation, flags: %x" , flags); }/*
@@ -612,8 +612,10 @@ static int mfill_atomic_pte_continue(strstruct page *page;int ret; -if (!ops) +if (!ops) { +VM_WARN_ONCE (1 ,"UFFDIO_CONTINUE for unsupported VMA" );return -EOPNOTSUPP ; + } folio = ops->get_folio_noalloc (inode, pgoff);/* Our caller expects us to return -EFAULT if we failed to find folio */
@@ -864,6 +866,7 @@ static __always_inline ssize_t mfill_atoif (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE ))return mfill_atomic_pte_zeropage (state); +VM_WARN_ONCE (1 ,"Unknown UFFDIO operation, flags: %x" , flags);return -EOPNOTSUPP ; }
@@ -2044,8 +2047,9 @@ bool vma_can_userfault(struct vm_area_streturn false;/* -* File backed memory with PTE level mappigns must implement -* ops->get_folio_noalloc() +* File backed VMAs (except HugeTLB) must implement +* ops->get_folio_noalloc() because it's required by __do_userfault() +* in page fault handling. */ if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && !ops->get_folio_noalloc)
_