mm, kvm: allow uffd suppot in guest_memfd
Subsystems (10)
Mike Rapoport
maintainercover mm, kvm: allow uffd suppot in guest_memfd
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in guest_memfd. They are quite different from the latest posting [1] so I'm restarting the versioning. As there was a lot of tension around the topic, this is an RFC to get some feedback and see how we can move forward.
As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.
In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.
Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.
Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.
[1] https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (12):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_get_pmd() helper.
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic() remove retry logic
mm: introduce VM_FAULT_UFFD_MINOR fault reason
Nikita Kalyazin (5):
mm: introduce VM_FAULT_UFFD_MISSING fault reason
KVM: guest_memfd: implement userfaultfd minor mode
KVM: guest_memfd: implement userfaultfd missing mode
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd
include/linux/mm.h | 5 + include/linux/mm_types.h | 15 +- include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 74 +- mm/hugetlb.c | 21 + mm/memory.c | 8 +- mm/shmem.c | 188 +++-- mm/userfaultfd.c | 671 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 134 +++- 10 files changed, 871 insertions(+), 450 deletions(-)
base-commit: f8f9c1f4d0c7a64600e2ca312dec824a0bc2f1da --
2.51.0
Mike Rapoport
maintainerRFC PATCH 1/17 userfaultfd: introduce mfill_copy_folio_locked() helper
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().
This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.
No functional change.
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/* +* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A +* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr); + +if (ret) +return -EFAULT ; + +flush_dcache_folio (folio); +return ret; +} + static int mfill_atomic_pte_copy(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,uffd_flags_t flags, struct folio **foliop) { - void *kaddr; int ret;struct folio *folio;
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,if (!folio) goto out; - kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (kaddr, (const void __user *) src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (kaddr); + ret =mfill_copy_folio_locked (folio, src_addr);/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* don't free the page */ goto out; } - -flush_dcache_folio (folio); }else { folio = *foliop; *foliop =NULL ;
Signature · 1 line
2.51.0
Re: [PATCH RFC 01/17] userfaultfd: introduce mfill_copy_folio_locked() helper
Quoted 11 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from
mfill_atomic_pte_copy()into a helper functionmfill_copy_folio_locked().This makes improves code readability and makes complex
mfill_atomic_pte_copy()function easier to comprehend.No functional change.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
The movement looks all fine,
Just one pure question to ask.
Quoted 19 lines
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)mm/userfaultfd.c +7
-0@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/*
mm/userfaultfd.c int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 257
Quoted 10 lines
mm/userfaultfd.c +10
-0+* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A
While moving, I wonder if we need this complex use case to describe the deadlock. Shouldn't this already happen with 1 process only?
process A thread 1 takes read lock (e.g. reaching here but
before copy_from_user)
process A thread 2 calls mmap, blocks taking write lock process A thread 1 goes on copy_from_user(), trigger page fault,
then tries to re-take the read lock
IIUC above should already cause deadlock when rwsem prioritize the write lock here.
Quoted 69 lines
mm/userfaultfd.c +18
-24+* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr); + +if (ret) +return -EFAULT ; + +flush_dcache_folio (folio); +return ret; +} + static int mfill_atomic_pte_copy(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,uffd_flags_t flags, struct folio **foliop) { - void *kaddr; int ret;struct folio *folio;@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,if (!folio) goto out; - kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (kaddr, (const void __user *) src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (kaddr); + ret =mfill_copy_folio_locked (folio, src_addr);/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) {@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* don't free the page */ goto out; } - -flush_dcache_folio (folio); }else { folio = *foliop; *foliop =NULL ;Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 01/17] userfaultfd: introduce mfill_copy_folio_locked() helper
Hi Peter,
Quoted 9 lines · Peter Xu
Quoted 8 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from
mfill_atomic_pte_copy()into a helper functionmfill_copy_folio_locked().This makes improves code readability and makes complex
mfill_atomic_pte_copy()function easier to comprehend.
Quoted 7 lines
Quoted 3 lines
No functional change.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>The movement looks all fine,
Acked-by Peter Xu <[email protected]>
Thanks!
Quoted 35 lines
Just one pure question to ask.
Quoted 19 lines
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)mm/userfaultfd.c +7
-0@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/* mm/userfaultfd.c int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 257
Quoted 10 lines
mm/userfaultfd.c +10
-0+* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A While moving, I wonder if we need this complex use case to describe the deadlock. Shouldn't this already happen with 1 process only?
Quoted 8 lines
process A thread 1 takes read lock (e.g. reaching here but
beforecopy_from_user)
process A thread 2 calls mmap, blocks taking write lock process A thread 1 goes oncopy_from_user(), trigger page fault,
then tries to re-take the read lockIIUC above should already cause deadlock when rwsem prioritize the write lock here.
We surely can improve the description here, but it should be a separate patch with its own changelog and it's out of scope of this series.
Quoted 9 lines
Quoted 9 lines
mm/userfaultfd.c +9
-0+* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr);
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerRFC PATCH 2/17 userfaultfd: introduce struct mfill_state
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic() passes a lot of parameters down to its callees.
Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.
Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.
The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.
--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)
mm/userfaultfd.c +82 -66
@@ -20,6 +20,20 @@#include "internal.h" #include "swap.h" +struct mfill_state { +struct userfaultfd_ctx *ctx; +unsigned long src_start; +unsigned long dst_start; +unsigned long len; +uffd_flags_t flags; + +struct vm_area_struct *vma; +unsigned long src_addr; +unsigned long dst_addr; +struct folio *folio; +pmd_t *pmd; +}; +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_atomic_pte_copy (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +unsigned long src_addr = state->src_addr; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio; +int ret; -if (!*foliop) { +if (!state->folio) { ret = -ENOMEM ; folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) { ret = -ENOENT ; - *foliop = folio; + state->folio = folio;/* don't free the page */ goto out; } }else { - folio = *foliop; - *foliop =NULL ; + folio = state->folio; + state->folio =NULL ; }/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,return ret; } -static int mfill_atomic_pte_zeropage (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_zeropage (struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl;int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, }/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ -static int mfill_atomic_pte_continue (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_continue (struct mfill_state *state) { -struct inode *inode =file_inode (dst_vma->vm_file); +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct inode *inode =file_inode (dst_vma->vm_file); +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio;struct page *page;int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ -static int mfill_atomic_pte_poison (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_poison (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma;struct mm_struct *dst_mm = dst_vma->vm_mm; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl; +int ret; _dst_pte =make_pte_marker (PTE_MARKER_POISONED ); ret = -EAGAIN ;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, uffd_flags_t flags);#endif /* CONFIG_HUGETLB_PAGE */ -static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long src_addr = state->src_addr; +unsigned long dst_addr = state->dst_addr; +struct folio **foliop = &state->folio; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;ssize_t err; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { -return mfill_atomic_pte_continue (dst_pmd, dst_vma, - dst_addr, flags); - }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) { -return mfill_atomic_pte_poison (dst_pmd, dst_vma, - dst_addr, flags); - } +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) +return mfill_atomic_pte_continue (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) +return mfill_atomic_pte_poison (state);/* * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,*/ if (!(dst_vma->vm_flags &VM_SHARED )) {if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); + err =mfill_atomic_pte_copy (state);else - err =mfill_atomic_pte_zeropage (dst_pmd, - dst_vma, dst_addr); + err =mfill_atomic_pte_zeropage (state); }else { err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long len, uffd_flags_t flags) { +struct mfill_state state = (struct mfill_state ){ + .ctx = ctx, + .dst_start = dst_start, + .src_start = src_start, + .flags = flags, + + .src_addr = src_start, + .dst_addr = dst_start, + };struct mm_struct *dst_mm = ctx->mm;struct vm_area_struct *dst_vma; +long copied =0 ;ssize_t err;pmd_t *dst_pmd; -unsigned long src_addr, dst_addr; -long copied; -struct folio *folio;/* * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); - src_addr = src_start; - dst_addr = dst_start; - copied =0 ; - folio =NULL ; retry:/* * Make sure the vma is not shared, that the dst range is
@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; -while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma; -VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); + +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;
@@ -827,34 +843,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,* tables under us; pte_offset_map_lock() will deal with that. */ - err =mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, - src_addr, flags, &folio); + state.pmd = dst_pmd; + err =mfill_atomic_pte (&state);cond_resched ();if (unlikely (err == -ENOENT )) {void *kaddr;up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); -VM_WARN_ON_ONCE (!folio); +uffd_mfill_unlock (state.vma); +VM_WARN_ON_ONCE (!state.folio); - kaddr =kmap_local_folio (folio,0 ); + kaddr =kmap_local_folio (state.folio,0 ); err =copy_from_user (kaddr, - (const void __user *) src_addr, + (const void __user *)state.src_addr,PAGE_SIZE );kunmap_local (kaddr);if (unlikely (err)) { err = -EFAULT ; goto out; } -flush_dcache_folio (folio); +flush_dcache_folio (state.folio); goto retry; }else -VM_WARN_ON_ONCE (folio); +VM_WARN_ON_ONCE (state.folio);if (!err) { - dst_addr +=PAGE_SIZE ; - src_addr +=PAGE_SIZE ; + state.dst_addr +=PAGE_SIZE ; + state.src_addr +=PAGE_SIZE ; copied +=PAGE_SIZE ;if (fatal_signal_pending (current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 3/17 userfaultfd: introduce mfill_get_pmd() helper.
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.
Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().
--- mm/userfaultfd.c | 103 ++++++++++++++++++++++++----------------------- 1 file changed, 53 insertions(+), 50 deletions(-)
mm/userfaultfd.c +53 -50
@@ -157,6 +157,57 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) +{ +pgd_t *pgd; +p4d_t *p4d; +pud_t *pud; + + pgd =pgd_offset (mm, address); + p4d =p4d_alloc (mm, pgd, address); +if (!p4d) +return NULL ; + pud =pud_alloc (mm, p4d, address); +if (!pud) +return NULL ; +/* +* Note that we didn't run this because the pmd was +* missing, the *pmd may be already established and in +* turn it may also be a trans_huge_pmd. +*/ +return pmd_alloc (mm, pud, address); +} + +static int mfill_get_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd; +pmd_t dst_pmdval; + + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +if (unlikely (pmd_none (dst_pmdval)) && +unlikely (__pte_alloc (dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_trans_huge (dst_pmdval))) +return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +} +/* Check if dst_addr is outside of file's size. Must be called with ptl held. */ static bool mfill_file_over_size (struct vm_area_struct *dst_vma,unsigned long dst_addr)
@@ -489,27 +540,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)return ret; } -static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) -{ -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; - - pgd =pgd_offset (mm, address); - p4d =p4d_alloc (mm, pgd, address); -if (!p4d) -return NULL ; - pud =pud_alloc (mm, p4d, address); -if (!pud) -return NULL ; -/* -* Note that we didn't run this because the pmd was -* missing, the *pmd may be already established and in -* turn it may also be a trans_huge_pmd. -*/ -return pmd_alloc (mm, pud, address); -} -#ifdef CONFIG_HUGETLB_PAGE /* * mfill_atomic processing for HUGETLB vmas. Note that this routine is
@@ -742,7 +772,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err; -pmd_t *dst_pmd;/* * Sanitize the command parameters:
@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_get_pmd (&state); +if (err)break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -if (unlikely (pmd_none (dst_pmdval)) && -unlikely (__pte_alloc (dst_mm, dst_pmd))) { - err = -ENOMEM ; -break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -/* -* If the dst_pmd is THP don't override it and just be strict. -* (This includes the case where the PMD used to be THP and -* changed back to none after __pte_alloc().) -*/ -if (unlikely (!pmd_present (dst_pmdval) || -pmd_trans_huge (dst_pmdval))) { - err = -EEXIST ; -break ; - } -if (unlikely (pmd_bad (dst_pmdval))) { - err = -EFAULT ; -break ; - }/* * For shmem mappings, khugepaged is allowed to remove page * tables under us; pte_offset_map_lock() will deal with that. */ - state.pmd = dst_pmd; err =mfill_atomic_pte (&state);cond_resched ();
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 4/17 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.
This function will be used later during refactoring of mfill_atomic_pte_copy().
Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.
--- mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 51 deletions(-)
mm/userfaultfd.c +73 -51
@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) + goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) + goto out; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock; + +out: + state->vma = dst_vma; +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;
@@ -768,8 +835,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, .src_addr = src_start, .dst_addr = dst_start, }; -struct mm_struct *dst_mm = ctx->mm; -struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err;
@@ -784,57 +849,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (dst_start + len <= dst_start); retry: -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ - dst_vma =uffd_mfill_lock (dst_mm, dst_start, len); -if (IS_ERR (dst_vma)) { - err =PTR_ERR (dst_vma); + err =mfill_get_vma (&state); +if (err) goto out; - } - -/* -* If memory mappings are changing because of non-cooperative -* operation (e.g. mremap) running in parallel, bail out and -* request the user to retry later -*/ -down_read (&ctx->map_changing_lock); - err = -EAGAIN ; -if (atomic_read (&ctx->mmap_changing)) - goto out_unlock; - - err = -EINVAL ; -/* -* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but -* it will overwrite vm_ops, so vma_is_anonymous must return false. -*/ -if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && - dst_vma->vm_flags &VM_SHARED )) - goto out_unlock; - -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) - goto out_unlock;/* * If this is a HUGETLB vma, pass off to appropriate routine */ -if (is_vm_hugetlb_page (dst_vma)) -return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, +if (is_vm_hugetlb_page (state.vma)) +return mfill_atomic_hugetlb (ctx, state.vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; - - state.vma = dst_vma; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);
@@ -853,8 +878,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (unlikely (err == -ENOENT )) {void *kaddr; -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state);VM_WARN_ON_ONCE (!state.folio); kaddr =kmap_local_folio (state.folio,0 );
@@ -883,9 +907,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -out_unlock: -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);
Signature · 1 line
2.51.0
Peter Xu
reviewerRe: [PATCH RFC 04/17] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Hi, Mike,
Quoted 71 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from
mfill_atomic()into a helper function.This function will be used later during refactoring of
mfill_atomic_pte_copy().Add a counterpart
mfill_put_vma()helper that unlocks the VMA and releasesmap_changing_lock.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 51 deletions(-)mm/userfaultfd.c +47
-0@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP ))
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 216
Quoted 10 lines
mm/userfaultfd.c +10
-0+ goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) + goto out; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock;
IMHO it's a bit weird to check for vma permissions in a get_vma() function.
Also, in the follow up patch it'll be also reused in mfill_copy_folio_retry() which doesn't need to check vma permission.
Maybe we can introduce mfill_vma_check() for these two checks? Then we can also drop the slightly weird is_vm_hugetlb_page() check (and "out" label) above.
Quoted 108 lines
mm/userfaultfd.c +16
-51+ +out: + state->vma = dst_vma; +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;@@ -768,8 +835,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, .src_addr = src_start, .dst_addr = dst_start, }; -struct mm_struct *dst_mm = ctx->mm; -struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err;@@ -784,57 +849,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (dst_start + len <= dst_start); retry: -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ - dst_vma =uffd_mfill_lock (dst_mm, dst_start, len); -if (IS_ERR (dst_vma)) { - err =PTR_ERR (dst_vma); + err =mfill_get_vma (&state); +if (err) goto out; - } - -/* -* If memory mappings are changing because of non-cooperative -* operation (e.g. mremap) running in parallel, bail out and -* request the user to retry later -*/ -down_read (&ctx->map_changing_lock); - err = -EAGAIN ; -if (atomic_read (&ctx->mmap_changing)) - goto out_unlock; - - err = -EINVAL ; -/* -* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but -* it will overwrite vm_ops, so vma_is_anonymous must return false. -*/ -if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && - dst_vma->vm_flags &VM_SHARED )) - goto out_unlock; - -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) - goto out_unlock;/* * If this is a HUGETLB vma, pass off to appropriate routine */ -if (is_vm_hugetlb_page (dst_vma)) -return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, +if (is_vm_hugetlb_page (state.vma)) +return mfill_atomic_hugetlb (ctx, state.vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; - - state.vma = dst_vma; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);@@ -853,8 +878,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (unlikely (err == -ENOENT )) {void *kaddr; -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state);VM_WARN_ON_ONCE (!state.folio); kaddr =kmap_local_folio (state.folio,0 );@@ -883,9 +907,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -out_unlock: -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 04/17] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Hi Peter,
Quoted 87 lines · Peter Xu
Hi, Mike,
Quoted 71 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from
mfill_atomic()into a helper function.This function will be used later during refactoring of
mfill_atomic_pte_copy().Add a counterpart
mfill_put_vma()helper that unlocks the VMA and releasesmap_changing_lock.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 51 deletions(-)mm/userfaultfd.c +47
-0@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP ))mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 216
Quoted 10 lines
mm/userfaultfd.c +10
-0+ goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) + goto out; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock;IMHO it's a bit weird to check for vma permissions in a
get_vma()function.
Quoted 6 lines
Also, in the follow up patch it'll be also reused in
mfill_copy_folio_retry()which doesn't need to check vma permission.Maybe we can introduce
mfill_vma_check()for these two checks? Then we can also drop the slightly weirdis_vm_hugetlb_page()check (and "out" label) above.
This version of get_vma() keeps the checks exactly as they were when we were retrying after dropping the lock and I prefer to have them this way to begin with.
Later we can optimize this further after the dust settles after these changes.
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerRFC PATCH 5/17 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of UFFDIO_COPY for anonymous memory might fail to copy data data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).
In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.
Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.
shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted afterwards.
--- mm/userfaultfd.c | 70 +++++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 24 deletions(-)
mm/userfaultfd.c +46 -24
@@ -404,35 +404,57 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_get_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) {/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto out_release; }/*
@@ -442,17 +464,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio); - ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;folio_put (folio); goto out; }
@@ -907,7 +928,8 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -mfill_put_vma (&state); +if (state.vma) +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);
Signature · 1 line
2.51.0
Peter Xu
reviewerRe: [PATCH RFC 05/17] userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
Hi, Mike,
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of
UFFDIO_COPYfor anonymous memory might fail to copy data data from userspace buffer when the destination VMA is locked (either withmm_lockor with per-VMA lock).In that case,
mfill_atomic()releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.
Quoted 7 lines
Since this retry-reget dance is only relevant for
UFFDIO_COPYand it never happens for otherUFFDIO_operations, make it a part ofmfill_atomic_pte_copy()that actually implementsUFFDIO_COPYfor anonymous memory.shmem implementation will be updated later and the loop in
mfill_atomic()will be adjusted afterwards.
Thanks for the refactoring. Looks good to me in general, only some nitpicks inline.
Quoted 71 lines
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 70 +++++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 24 deletions(-)mm/userfaultfd.c +37
-9@@ -404,35 +404,57 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_get_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) {/* fallback to copy_from_user outside mmap_lock */
mm/userfaultfd.c static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) · line 455
Quoted 10 lines
mm/userfaultfd.c +1
-9-if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_retry (state, folio);
Yes, I agree this should work and should avoid the previous ENOENT processing that might be hard to follow. It'll move the complexity into mfill_state though (e.g., now it's unknown on the vma lock state after this function returns..), but I guess it's fine.
mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 465
Quoted 8 lines
mm/userfaultfd.c +2
-0+if (ret) + goto out_release; }/* @@ -442,17 +464,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio);
Since success path should make sure vma lock held when reaching here, but now with mfill_copy_folio_retry()'s presence it's not as clear as before, maybe we add an assertion for that here before installing ptes? No strong feelings.
Quoted 6 lines
mm/userfaultfd.c +0
-5- ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 476
Quoted 10 lines
mm/userfaultfd.c +4
-0+ ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;
I recall the code removed is the only path that can return ENOENT? Then maybe this line isn't needed?
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 931
Quoted 9 lines
mm/userfaultfd.c +1
-1folio_put (folio); goto out; }@@ -907,7 +928,8 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -mfill_put_vma (&state); +if (state.vma)
I wonder if we should move this check into mfill_put_vma() directly, it might be overlooked if we'll put_vma in other paths otherwise.
Quoted 7 lines
mm/userfaultfd.c +1
-0+mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 05/17] userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
Quoted 102 lines · Peter Xu
Hi, Mike,
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of
UFFDIO_COPYfor anonymous memory might fail to copy data data from userspace buffer when the destination VMA is locked (either withmm_lockor with per-VMA lock).In that case,
mfill_atomic()releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.Quoted 7 lines
Since this retry-reget dance is only relevant for
UFFDIO_COPYand it never happens for otherUFFDIO_operations, make it a part ofmfill_atomic_pte_copy()that actually implementsUFFDIO_COPYfor anonymous memory.shmem implementation will be updated later and the loop in
mfill_atomic()will be adjusted afterwards.Thanks for the refactoring. Looks good to me in general, only some nitpicks inline.
Quoted 78 lines
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 70 +++++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 24 deletions(-)mm/userfaultfd.c +37
-16@@ -404,35 +404,57 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_get_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) {/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else {
Quoted 8 lines
mm/userfaultfd.c static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) · line 455
Quoted 3 lines
mm/userfaultfd.c +1
-2- folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_retry (state, folio);Yes, I agree this should work and should avoid the previous ENOENT processing that might be hard to follow. It'll move the complexity into
mfill_statethough (e.g., now it's unknown on the vma lock state after this function returns..), but I guess it's fine.
When this function returns success VMA is locked. If the function fails it does not matter if the VMA is locked.
I'll add some comments.
Quoted 4 lines
Quoted 4 lines
mm/userfaultfd.c +2
-0+if (ret) + goto out_release; }
Quoted 9 lines
mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 465
Quoted 4 lines
mm/userfaultfd.c +0
-0/* @@ -442,17 +464,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio);Since success path should make sure vma lock held when reaching here, but now with mfill_copy_folio_retry()'s presence it's not as clear as before, maybe we add an assertion for that here before installing ptes? No strong feelings.
I'll add comments.
Quoted 11 lines
Quoted 11 lines
mm/userfaultfd.c +1
-5- ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:
Quoted 8 lines
mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 476
Quoted 5 lines
mm/userfaultfd.c +3
-0return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;I recall the code removed is the only path that can return ENOENT? Then maybe this line isn't needed?
I didn't want to audit all potential errors and this is a temporal safety measure to avoid breaking biscection. This is anyway removed in the following patches.
Quoted 7 lines
Quoted 7 lines
mm/userfaultfd.c +0
-0folio_put (folio); goto out; }@@ -907,7 +928,8 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; }
Quoted 5 lines
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 931
Quoted 2 lines
mm/userfaultfd.c +1
-1-mfill_put_vma (&state); +if (state.vma)I wonder if we should move this check into
mfill_put_vma()directly, it might be overlooked if we'llput_vmain other paths otherwise.
Yeah, I'll check this.
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerRFC PATCH 6/17 userfaultfd: move vma_can_userfault out of line
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault() has grown pretty big and it's not called on performance critical path.
Move it out of line.
No functional changes.
--- include/linux/userfaultfd_k.h | 35 ++--------------------------------- mm/userfaultfd.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags, -bool wp_async) -{ - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true; - -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && - !vma_is_anonymous (vma)) -return false; - -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async);static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {
mm/userfaultfd.c +33 -0
@@ -2016,6 +2016,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,return moved ? moved : err; } +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async) +{ + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + +if ((vm_flags &VM_UFFD_MINOR ) && + (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) +return false; + +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem & hugetlbfs are not supported but only +* anonymous. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && + !vma_is_anonymous (vma)) +return false; + +/* By default, allow any of anon|shmem|hugetlb */ +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +static void userfaultfd_set_vm_flags (struct vm_area_struct *vma,vm_flags_t vm_flags) {
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 7/17 userfaultfd: introduce vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.
Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.
Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.
Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.
--- include/linux/mm.h | 5 +++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 21 ++++++++++++++++++ mm/shmem.c | 23 ++++++++++++++++++++ mm/userfaultfd.c | 41 ++++++++++++++++++++++------------- 5 files changed, 81 insertions(+), 15 deletions(-)
include/linux/mm.h +5 -0
@@ -732,6 +732,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -817,6 +819,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;
mm/hugetlb.c +21 -0
@@ -4797,6 +4797,24 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then hugetlb is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false; +return true; +} + +static const struct vm_uffd_ops hugetlb_uffd_ops = { + .can_userfault = hugetlb_can_userfault, +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4810,6 +4828,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +23 -0
@@ -5290,6 +5290,23 @@ static const struct super_operations shmem_ops = {#endif }; +#ifdef CONFIG_USERFAULTFD +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false; +return true; +} + +static const struct vm_uffd_ops shmem_uffd_ops = { + .can_userfault = shmem_can_userfault, +}; +#endif +static const struct vm_operations_struct shmem_vm_ops = { .fault = shmem_fault, .map_pages = filemap_map_pages,
@@ -5297,6 +5314,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5306,6 +5326,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)
mm/userfaultfd.c +26 -15
63 lines not on this page · raw message
Signature · 1 line
2.51.0
Peter Xu
reviewerRe: [PATCH RFC 07/17] userfaultfd: introduce vm_uffd_ops
Quoted 84 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on
vma_is_anonymous(),vma_is_shmem()andis_vm_hugetlb_page()and some of these conditions can be expressed as operations implemented by a particular memory type.Introduce
vm_uffd_opsextension tovm_operations_structthat will delegate memory type specific operations to a VMA owner.Operations for anonymous memory are handled internally in userfaultfd using
anon_uffd_opsthat implicitly assigned to anonymous VMAs.Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of
vma_can_userfault()into the new callbacks.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/mm.h | 5 +++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 21 ++++++++++++++++++ mm/shmem.c | 23 ++++++++++++++++++++ mm/userfaultfd.c | 41 ++++++++++++++++++++++------------- 5 files changed, 81 insertions(+), 15 deletions(-)include/linux/mm.h +5
-0@@ -732,6 +732,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer @@ -817,6 +819,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING include/linux/userfaultfd_k.h +6
-0@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;mm/hugetlb.c +0
-0@@ -4797,6 +4797,24 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; }
mm/hugetlb.c static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf) · line 4809
Quoted 10 lines
mm/hugetlb.c +10
-0+#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then hugetlb is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false;
IMHO we don't need to dup this for every vm_uffd_ops driver. It might be unnecessary to even make driver be aware how pte marker plays the role here, because pte markers are needed for all page cache file systems anyway. There should have no outliers. Instead we can just let can_userfault() report whether the driver generically supports userfaultfd, leaving the detail checks for core mm.
I understand you wanted to also make anon to be a driver, so this line won't apply to anon. However IMHO anon is special enough so we can still make this in the generic path.
Quoted 142 lines
mm/hugetlb.c +11
-0+ return true; +} + +static const struct vm_uffd_ops hugetlb_uffd_ops = { + .can_userfault = hugetlb_can_userfault, +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops. @@ -4810,6 +4828,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,mm/shmem.c +23
-0@@ -5290,6 +5290,23 @@ static const struct super_operations shmem_ops = {#endif }; +#ifdef CONFIG_USERFAULTFD +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false; +return true; +} + +static const struct vm_uffd_ops shmem_uffd_ops = { + .can_userfault = shmem_can_userfault, +}; +#endif +static const struct vm_operations_struct shmem_vm_ops = { .fault = shmem_fault, .map_pages = filemap_map_pages,@@ -5297,6 +5314,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {@@ -5306,6 +5326,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)mm/userfaultfd.c +26
-15@@ -34,6 +34,25 @@ struct mfill_state {pmd_t *pmd; }; +static bool anon_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +/* anonymous memory does not support MINOR mode */ +if (vm_flags &VM_UFFD_MINOR ) +return false; +return true; +} + +static const struct vm_uffd_ops anon_uffd_ops = { + .can_userfault = anon_can_userfault, +}; + +static const struct vm_uffd_ops *vma_uffd_ops (struct vm_area_struct *vma) +{ +if (vma_is_anonymous (vma)) +return &anon_uffd_ops; +return vma->vm_ops ? vma->vm_ops->uffd_ops :NULL ; +} +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {@@ -2019,13 +2038,15 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags,bool wp_async) { - vm_flags &= __VM_UFFD_FLAGS; +const struct vm_uffd_ops *ops =vma_uffd_ops (vma); -if (vma->vm_flags &VM_DROPPABLE ) +/* only VMAs that implement vm_uffd_ops are supported */ +if (!ops)return false; -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE )return false;/* @@ -2035,18 +2056,8 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,if (wp_async && (vm_flags ==VM_UFFD_WP ))return true; -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && - !vma_is_anonymous (vma)) -return false; -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); +return ops->can_userfault (vma, vm_flags); } static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 07/17] userfaultfd: introduce vm_uffd_ops
Hi Peter,
Quoted 103 lines · Peter Xu
Quoted 84 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on
vma_is_anonymous(),vma_is_shmem()andis_vm_hugetlb_page()and some of these conditions can be expressed as operations implemented by a particular memory type.Introduce
vm_uffd_opsextension tovm_operations_structthat will delegate memory type specific operations to a VMA owner.Operations for anonymous memory are handled internally in userfaultfd using
anon_uffd_opsthat implicitly assigned to anonymous VMAs.Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of
vma_can_userfault()into the new callbacks.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/mm.h | 5 +++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 21 ++++++++++++++++++ mm/shmem.c | 23 ++++++++++++++++++++ mm/userfaultfd.c | 41 ++++++++++++++++++++++------------- 5 files changed, 81 insertions(+), 15 deletions(-)include/linux/mm.h +5
-0@@ -732,6 +732,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer @@ -817,6 +819,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING include/linux/userfaultfd_k.h +6
-0@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;mm/hugetlb.c +0
-0@@ -4797,6 +4797,24 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; }mm/hugetlb.c static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf) · line 4809
Quoted 10 lines
mm/hugetlb.c +10
-0+#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then hugetlb is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false;IMHO we don't need to dup this for every
vm_uffd_opsdriver. It might be unnecessary to even make driver be aware how pte marker plays the role here, because pte markers are needed for all page cache file systems anyway. There should have no outliers. Instead we can just letcan_userfault()report whether the driver generically supports userfaultfd, leaving the detail checks for core mm.
Quoted 3 lines
I understand you wanted to also make anon to be a driver, so this line won't apply to anon. However IMHO anon is special enough so we can still make this in the generic path.
Well, the idea is to drop all vma_is*() in can_userfault(). And maybe eventually in entire mm/userfaultfd.c
If all page cache filesystems need this, something like this should work, right?
if (!uffd_supports_wp_marker() && (vma->vm_flags & VM_SHARED) && (vm_flags & VM_UFFD_WP)) return false;
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH RFC 07/17] userfaultfd: introduce vm_uffd_ops
Quoted 113 lines · Mike Rapoport
Hi Peter,
Quoted 103 lines · Peter Xu
Quoted 84 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on
vma_is_anonymous(),vma_is_shmem()andis_vm_hugetlb_page()and some of these conditions can be expressed as operations implemented by a particular memory type.Introduce
vm_uffd_opsextension tovm_operations_structthat will delegate memory type specific operations to a VMA owner.Operations for anonymous memory are handled internally in userfaultfd using
anon_uffd_opsthat implicitly assigned to anonymous VMAs.Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of
vma_can_userfault()into the new callbacks.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/mm.h | 5 +++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 21 ++++++++++++++++++ mm/shmem.c | 23 ++++++++++++++++++++ mm/userfaultfd.c | 41 ++++++++++++++++++++++------------- 5 files changed, 81 insertions(+), 15 deletions(-)include/linux/mm.h +5
-0@@ -732,6 +732,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer @@ -817,6 +819,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING include/linux/userfaultfd_k.h +6
-0@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;mm/hugetlb.c +0
-0@@ -4797,6 +4797,24 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; }mm/hugetlb.c static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf) · line 4809
Quoted 10 lines
mm/hugetlb.c +10
-0+#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then hugetlb is not supported. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP )) +return false;IMHO we don't need to dup this for every
vm_uffd_opsdriver. It might be unnecessary to even make driver be aware how pte marker plays the role here, because pte markers are needed for all page cache file systems anyway. There should have no outliers. Instead we can just letcan_userfault()report whether the driver generically supports userfaultfd, leaving the detail checks for core mm.Quoted 3 lines
I understand you wanted to also make anon to be a driver, so this line won't apply to anon. However IMHO anon is special enough so we can still make this in the generic path.
Well, the idea is to drop all vma_is*() in
can_userfault(). And maybe eventually in entiremm/userfaultfd.c
Quoted 6 lines
If all page cache filesystems need this, something like this should work, right?
if (!uffd_supports_wp_marker() && (vma->vm_flags & VM_SHARED) && (vm_flags & VM_UFFD_WP)) return false;
Sorry for a late response.
IIUC we can't check against VM_SHARED, because we need pte markers also for MAP_PRIVATE on file mappings.
The need of pte markers come from the fact that the vma has a page cache backing it, rather than whether it's a shared or private mapping. Consider if a file mapping vma + MAP_PRIVATE, if we wr-protect the vma with nothing populated, we want to still get notified whenever there's a write.
So the original check should be good.
I'm fine with most of the rest comments in this series I left and I'm OK if you prefer settle things down first. For this one, I still want to see if we can move this to uffd core code.
The whole point is I want to have zero info leaked about pte marker into module ops.
For that, IMHO it'll be fine we use one vma_is_anonymous() is uffd core code once.
Actually, I don't think uffd core can get rid of handling anon specially. With this series applied, mfill_atomic_pte_copy() will still need to hard-code anon processing on MAP_PRIVATE and I don't think it can go away..
mfill_atomic_pte_copy():
if (!(state->vma->vm_flags & VM_SHARED)) ops = &anon_uffd_ops;
IMHO using vma_is_anonymous() for one more time should be better than leaking pte marker whole concept to modules. So the driver should only report if the driver supports UFFD_WP in general. It shouldn't care about anything the core mm would already do otherwise, including this one on "whether system config / arch has globally enabled pte markers" and the relation between that config and the WP feature impl details.
Thanks,
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 07/17] userfaultfd: introduce vm_uffd_ops
Quoted 16 lines · Peter Xu
Quoted 8 lines · Mike Rapoport
Quoted 4 lines
I understand you wanted to also make anon to be a driver, so this line won't apply to anon. However IMHO anon is special enough so we can still make this in the generic path.
Well, the idea is to drop all vma_is*() in
can_userfault(). And maybe eventually in entiremm/userfaultfd.cQuoted 6 lines
If all page cache filesystems need this, something like this should work, right?
if (!uffd_supports_wp_marker() && (vma->vm_flags & VM_SHARED) && (vm_flags & VM_UFFD_WP)) return false;
Quoted 8 lines
Sorry for a late response.
IMHO using
vma_is_anonymous()for one more time should be better than leaking pte marker whole concept to modules. So the driver should only report if the driver supportsUFFD_WPin general. It shouldn't care about anything the core mm would already do otherwise, including this one on "whether system config / arch has globally enabled pte markers" and the relation between that config and the WP feature impl details.
I agree. Will move the check for the markers back into userfaultfd.c
Quoted 5 lines
Thanks,
Signature · 1 line
Peter Xu
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH RFC 07/17] userfaultfd: introduce vm_uffd_ops
Quoted 1 line · Mike Rapoport
I agree. Will move the check for the markers back into
userfaultfd.c
Thank you!
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRFC PATCH 8/17 userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.
Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.
--- include/linux/userfaultfd_k.h | 7 +++++++ mm/shmem.c | 15 ++++++++++++++- mm/userfaultfd.c | 32 ++++++++++++++++---------------- 3 files changed, 37 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* +* Called to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +* The returned folio is locked and with reference held. +*/ +struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); };/* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
@@ -5291,6 +5291,18 @@ static const struct super_operations shmem_ops = { };#ifdef CONFIG_USERFAULTFD +static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +} +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) {/*
@@ -5303,7 +5315,8 @@ static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) }static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .can_userfault = shmem_can_userfault, + .get_folio_noalloc = shmem_get_folio_noalloc, };#endif
mm/userfaultfd.c +16 -16
@@ -188,6 +188,7 @@ static int mfill_get_vma(struct mfill_state *state)struct userfaultfd_ctx *ctx = state->ctx;uffd_flags_t flags = state->flags;struct vm_area_struct *dst_vma; +const struct vm_uffd_ops *ops;int err;/*
@@ -228,10 +229,12 @@ static int mfill_get_vma(struct mfill_state *state)if (is_vm_hugetlb_page (dst_vma)) goto out; -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + ops =vma_uffd_ops (dst_vma); +if (!ops) goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && + !ops->get_folio_noalloc) goto out_unlock; out:
@@ -568,6 +571,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)static int mfill_atomic_pte_continue (struct mfill_state *state) {struct vm_area_struct *dst_vma = state->vma; +const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);
@@ -577,16 +581,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (!ops) +return -EOPNOTSUPP ; + + folio = ops->get_folio_noalloc (inode, pgoff);/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; - } +if (IS_ERR_OR_NULL (folio)) +return -EFAULT ; page =folio_file_page (folio, pgoff);if (PageHWPoison (page)) {
@@ -600,13 +601,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 9/17 userfaultfd: introduce vm_uffd_ops->alloc_folio()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio
Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.
Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.
Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should return allocate a and return folio or NULL if allocation fails. +*/ +struct folio *(*alloc_folio)(struct vm_area_struct *vma, +unsigned long addr); };/* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44
137 lines not on this page · raw message
Signature · 1 line
2.51.0
Peter Xu
reviewerRe: [PATCH RFC 09/17] userfaultfd: introduce vm_uffd_ops->alloc_folio()
On Tue, Jan 27, 2026 at 09:29:28PM +0200, Mike Rapoport wrote:
[...]
Quoted 42 lines
mm/userfaultfd.c +8
-26-static int mfill_atomic_pte_zeroed_folio (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct folio *folio; -int ret = -ENOMEM ; - - folio =vma_alloc_zeroed_movable_folio (dst_vma, dst_addr); -if (!folio) -return ret; - -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_put; +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -/* -* The memory barrier inside __folio_mark_uptodate makes sure that -* zeroing out the folio become visible before mapping the page -* using set_pte_at(). See do_anonymous_page(). -*/ -__folio_mark_uptodate (folio); +return __mfill_atomic_pte (state, ops); +} - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true,0 ); -if (ret) - goto out_put; +static int mfill_atomic_pte_zeroed_folio (struct mfill_state *state) +{ +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -return 0 ; -out_put: -folio_put (folio); -return ret; +return __mfill_atomic_pte (state, ops); }static int mfill_atomic_pte_zeropage (struct mfill_state *state)
mm/userfaultfd.c static int mfill_atomic_pte_zeropage(struct mfill_state *state) · line 549
Quoted 5 lines
mm/userfaultfd.c +1
-1@@ -542,7 +546,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)int ret;if (mm_forbids_zeropage (dst_vma->vm_mm)) -return mfill_atomic_pte_zeroed_folio (dst_pmd, dst_vma, dst_addr); +return mfill_atomic_pte_zeroed_folio (state);
After this patch, mfill_atomic_pte_zeroed_folio() should be 100% the same impl with mfill_atomic_pte_copy(), so IIUC we can drop it.
Quoted 6 lines
mm/userfaultfd.c +0
-0_dst_pte =pte_mkspecial (pfn_pte (my_zero_pfn (dst_addr), dst_vma->vm_page_prot));Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 09/17] userfaultfd: introduce vm_uffd_ops->alloc_folio()
Quoted 48 lines · Peter Xu
On Tue, Jan 27, 2026 at 09:29:28PM +0200, Mike Rapoport wrote:
[...]
Quoted 44 lines
mm/userfaultfd.c +8
-26-static int mfill_atomic_pte_zeroed_folio (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct folio *folio; -int ret = -ENOMEM ; - - folio =vma_alloc_zeroed_movable_folio (dst_vma, dst_addr); -if (!folio) -return ret; - -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_put; +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -/* -* The memory barrier inside __folio_mark_uptodate makes sure that -* zeroing out the folio become visible before mapping the page -* using set_pte_at(). See do_anonymous_page(). -*/ -__folio_mark_uptodate (folio); +return __mfill_atomic_pte (state, ops); +} - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true,0 ); -if (ret) - goto out_put; +static int mfill_atomic_pte_zeroed_folio (struct mfill_state *state) +{ +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -return 0 ; -out_put: -folio_put (folio); -return ret; +return __mfill_atomic_pte (state, ops); }static int mfill_atomic_pte_zeropage (struct mfill_state *state)@@ -542,7 +546,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)int ret;
Quoted 6 lines
mm/userfaultfd.c static int mfill_atomic_pte_zeropage(struct mfill_state *state) · line 549
Quoted 3 lines
mm/userfaultfd.c +1
-1if (mm_forbids_zeropage (dst_vma->vm_mm)) -return mfill_atomic_pte_zeroed_folio (dst_pmd, dst_vma, dst_addr); +return mfill_atomic_pte_zeroed_folio (state);After this patch,
mfill_atomic_pte_zeroed_folio()should be 100% the same impl withmfill_atomic_pte_copy(), so IIUC we can drop it.
It will be slightly different after the next patch to emphasize that copying into MAP_PRIVATE actually creates anonymous memory.
Quoted 9 lines
Quoted 5 lines
mm/userfaultfd.c +0
-0_dst_pte =pte_mkspecial (pfn_pte (my_zero_pfn (dst_addr), dst_vma->vm_page_prot));Signature · 1 line
2.51.0Signature · 1 line
Peter Xu
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH RFC 09/17] userfaultfd: introduce vm_uffd_ops->alloc_folio()
Quoted 56 lines · Mike Rapoport
Quoted 48 lines · Peter Xu
On Tue, Jan 27, 2026 at 09:29:28PM +0200, Mike Rapoport wrote:
[...]
Quoted 44 lines
mm/userfaultfd.c +8
-26-static int mfill_atomic_pte_zeroed_folio (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct folio *folio; -int ret = -ENOMEM ; - - folio =vma_alloc_zeroed_movable_folio (dst_vma, dst_addr); -if (!folio) -return ret; - -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_put; +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -/* -* The memory barrier inside __folio_mark_uptodate makes sure that -* zeroing out the folio become visible before mapping the page -* using set_pte_at(). See do_anonymous_page(). -*/ -__folio_mark_uptodate (folio); +return __mfill_atomic_pte (state, ops); +} - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true,0 ); -if (ret) - goto out_put; +static int mfill_atomic_pte_zeroed_folio (struct mfill_state *state) +{ +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -return 0 ; -out_put: -folio_put (folio); -return ret; +return __mfill_atomic_pte (state, ops); }static int mfill_atomic_pte_zeropage (struct mfill_state *state)@@ -542,7 +546,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)int ret;Quoted 6 lines
mm/userfaultfd.c static int mfill_atomic_pte_zeropage(struct mfill_state *state) · line 549
Quoted 3 lines
mm/userfaultfd.c +1
-1if (mm_forbids_zeropage (dst_vma->vm_mm)) -return mfill_atomic_pte_zeroed_folio (dst_pmd, dst_vma, dst_addr); +return mfill_atomic_pte_zeroed_folio (state);After this patch,
mfill_atomic_pte_zeroed_folio()should be 100% the same impl withmfill_atomic_pte_copy(), so IIUC we can drop it.
Quoted 2 lines
It will be slightly different after the next patch to emphasize that copying into
MAP_PRIVATEactually creates anonymous memory.
True. It might be helpful to leave a line in the commit message so it's intentional to temporarily have two functions do the same thing, but I'm OK either way.
Thanks,
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRFC PATCH 10/17 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.
Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().
Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c
mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.
fixup
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 20 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 155 deletions(-)
include/linux/shmem_fs.h +0 -14
@@ -223,20 +223,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +15 -5
@@ -97,6 +97,21 @@ struct vm_uffd_ops {*/ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache. +* Returns 0 on success, error code on failre. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +* The folio should be unlocked, but the reference to it should not be +* dropped. +*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */
@@ -130,11 +145,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);
mm/shmem.c +53 -95
177 lines not on this page · raw message
mm/userfaultfd.c +38 -41
148 lines not on this page · raw message
Signature · 1 line
2.51.0
Peter Xu
reviewerRe: [PATCH RFC 10/17] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 9 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Add
filemap_add()andfilemap_remove()methods tovm_uffd_opsand use them in__mfill_atomic_pte()to add shmem folios to page cache and remove them in case of error.Implement these methods in shmem along with
vm_uffd_ops->alloc_folio()and dropshmem_mfill_atomic_pte().
Quoted 7 lines
Since userfaultfd now does not reference any functions from shmem, drop include if
linux/shmem_fs.hfrommm/userfaultfd.c
mfill_atomic_install_pte()is not used anywhere outside of mm/userfaultfd, make it static.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
This patch looks like a real nice cleanup on its own, thanks Mike!
I guess I never tried to read into shmem accountings, now after I read some of the codes I don't see any issue with your change. We can also wait for some shmem developers double check those. Comments inline below on something I spot.
Quoted 4 lines
fixup
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
[unexpected lines can be removed here]
Quoted 37 lines
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 20 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 155 deletions(-)include/linux/shmem_fs.h +0
-14@@ -223,20 +223,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that include/linux/userfaultfd_k.h +0
-0
include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 103
Quoted 7 lines
include/linux/userfaultfd_k.h +4
-0@@ -97,6 +97,21 @@ struct vm_uffd_ops {*/ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache. +* Returns 0 on success, error code on failre.
failure
include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 112
Quoted 9 lines
include/linux/userfaultfd_k.h +9
-0+*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +* The folio should be unlocked, but the reference to it should not be +* dropped.
Might be slightly misleading to explicitly mention this? As page cache also holds references and IIUC they need to be dropped there. But I get your point, on keeping the last refcount due to allocation.
IMHO the "should revert the operation of ->filemap_add()" is good enough and accurately describes it.
Quoted 146 lines
include/linux/userfaultfd_k.h +2
-5+*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */ @@ -130,11 +145,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);mm/shmem.c +27
-83@@ -3169,118 +3169,73 @@ static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap,#endif /* CONFIG_TMPFS_QUOTA */ #ifdef CONFIG_USERFAULTFD -int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) -{ -struct inode *inode =file_inode (dst_vma->vm_file); -struct shmem_inode_info *info =SHMEM_I (inode); +static struct folio *shmem_mfill_folio_alloc (struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file);struct address_space *mapping = inode->i_mapping; +struct shmem_inode_info *info =SHMEM_I (inode); +pgoff_t pgoff =linear_page_index (vma, addr);gfp_t gfp =mapping_gfp_mask (mapping); -pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); -void *page_kaddr;struct folio *folio; -int ret; -pgoff_t max_off; - -if (shmem_inode_acct_blocks (inode,1 )) { -/* -* We may have got a page, returned -ENOENT triggering a retry, -* and now we find ourselves with -ENOMEM. Release the page, to -* avoid a BUG_ON in our caller. -*/ -if (unlikely (*foliop)) { -folio_put (*foliop); - *foliop =NULL ; - } -return -ENOMEM ; - } -if (!*foliop) { - ret = -ENOMEM ; - folio =shmem_alloc_folio (gfp,0 , info, pgoff); -if (!folio) - goto out_unacct_blocks; +if (unlikely (pgoff >=DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ))) +return NULL ; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { - page_kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (page_kaddr, - (const void __user *)src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (page_kaddr); - -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - *foliop = folio; - ret = -ENOENT ; -/* don't free the page */ - goto out_unacct_blocks; - } + folio =shmem_alloc_folio (gfp,0 , info, pgoff); +if (!folio) +return NULL ; -flush_dcache_folio (folio); - }else {/* ZEROPAGE */ -clear_user_highpage (&folio->page, dst_addr); - } - }else { - folio = *foliop; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - *foliop =NULL ; +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) { +folio_put (folio); +return NULL ; } -VM_BUG_ON (folio_test_locked (folio)); -VM_BUG_ON (folio_test_swapbacked (folio)); +return folio; +} + +static int shmem_mfill_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +gfp_t gfp =mapping_gfp_mask (mapping); +int err; +__folio_set_locked (folio);__folio_set_swapbacked (folio); -__folio_mark_uptodate (folio); - - ret = -EFAULT ; - max_off =DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ); -if (unlikely (pgoff >= max_off)) - goto out_release; - ret =mem_cgroup_charge (folio, dst_vma->vm_mm, gfp); -if (ret) - goto out_release; - ret =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); -if (ret)
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3214
Quoted 10 lines
mm/shmem.c +4
-5- goto out_release; + err =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); +if (err) + goto err_unlock; - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true, flags); -if (ret) - goto out_delete_from_cache; +if (shmem_inode_acct_blocks (inode,1 )) {
We used to do this early before allocation, IOW, I think we still have an option to leave this to alloc_folio() hook. However I don't see an issue either keeping it in filemap_add(). Maybe this movement should better be spelled out in the commit message anyway on how this decision is made.
IIUC it's indeed safe we move this acct_blocks() here, I even see Hugh mentioned such in an older commit 3022fd7af96, but Hugh left uffd alone at that time:
Userfaultfd is a foreign country: they do things differently there, and for good reason - to avoid mmap_lock deadlock. Leave ordering in shmem_mfill_atomic_pte() untouched for now, but I would rather like to mesh it better with shmem_get_folio_gfp() in the future.
I'm not sure if that's also what you wanted to do - to make userfaultfd code work similarly like what shmem_alloc_and_add_folio() does right now. Maybe you want to mention that too somewhere in the commit log when posting a formal patch.
One thing not directly relevant is, shmem_alloc_and_add_folio() also does proper recalc of inode allocation info when acct_blocks() fails here. But if that's a problem, that's pre-existing for userfaultfd, so IIUC we can also leave it alone until someone (maybe quota user) complains about shmem allocation failures on UFFDIO_COPY.. It's just that it looks similar problem here in userfaultfd path.
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3219
Quoted 5 lines
mm/shmem.c +4
-0+ err = -ENOMEM ; + goto err_delete_from_cache; + } +folio_add_lru (folio);
This change is pretty separate from the work, but looks correct to me: IIUC we moved the lru add earlier now, and it should be safe as long as we're holding folio lock all through the process, and folio_put() (ultimately, __page_cache_release()) will always properly undo the lru change. Please help double check if my understanding is correct.
Quoted 15 lines
mm/shmem.c +9
-3shmem_recalc_inode (inode,1 ,0 ); -folio_unlock (folio); +return 0 ; -out_delete_from_cache: + +err_delete_from_cache:filemap_remove_folio (folio); -out_release: +err_unlock: +folio_unlock (folio); +return err; +} + +static void shmem_mfill_filemap_remove(struct folio *folio,
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3282
Quoted 10 lines
mm/shmem.c +6
-3+ struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); + +filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 );
This looks wrong, or maybe I miss somewhere we did the unacct_blocks()?
Quoted 52 lines
mm/shmem.c +3
-1- return ret; }#endif /* CONFIG_USERFAULTFD */ @@ -5317,6 +5272,9 @@ static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)static const struct vm_uffd_ops shmem_uffd_ops = { .can_userfault = shmem_can_userfault, .get_folio_noalloc = shmem_get_folio_noalloc, + .alloc_folio = shmem_mfill_folio_alloc, + .filemap_add = shmem_mfill_filemap_add, + .filemap_remove = shmem_mfill_filemap_remove, };#endif mm/userfaultfd.c +4
-8@@ -14,7 +14,6 @@#include <linux/userfaultfd_k.h> #include <linux/mmu_notifier.h> #include <linux/hugetlb.h> -#include <linux/shmem_fs.h> #include <asm/tlbflush.h> #include <asm/tlb.h> #include "internal.h" @@ -337,10 +336,10 @@ static bool mfill_file_over_size(struct vm_area_struct *dst_vma,* This function handles both MCOPY_ATOMIC_NORMAL and _CONTINUE for both shmem * and anon, and for both shared and private VMAs. */ -int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags) +static int mfill_atomic_install_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr,struct page *page, +uffd_flags_t flags) {int ret;struct mm_struct *dst_mm = dst_vma->vm_mm;@@ -384,9 +383,6 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, goto out_unlock;if (page_in_cache) { -/* Usually, cache pages are already added to LRU */ -if (newly_allocated) -folio_add_lru (folio);folio_add_file_rmap_pte (folio, page, dst_vma); }else {folio_add_new_anon_rmap (folio, dst_vma, dst_addr,RMAP_EXCLUSIVE );
mm/userfaultfd.c int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 401
Quoted 5 lines
mm/userfaultfd.c +2
-0@@ -401,6 +397,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte); +if (page_in_cache) +folio_unlock (folio);
Nitpick: another small change that looks correct, but IMHO would be nice to either make it a small separate patch, or mention in the commit message.
Quoted 102 lines
mm/userfaultfd.c +31
-21+/* No need to invalidate - it was non-present before */ update_mmu_cache (dst_vma, dst_addr, dst_pte); ret =0 ;@@ -507,13 +506,22 @@ static int __mfill_atomic_pte(struct mfill_state *state,*/ __folio_mark_uptodate (folio); +if (ops->filemap_add) { + ret = ops->filemap_add (folio, state->vma, state->dst_addr); +if (ret) + goto err_folio_put; + } + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, - &folio->page, true, flags); + &folio->page, flags);if (ret) - goto err_folio_put; + goto err_filemap_remove;return 0 ; +err_filemap_remove: +if (ops->filemap_remove) + ops->filemap_remove (folio, state->vma); err_folio_put:folio_put (folio);/* Don't return -ENOENT so that our caller won't retry */ @@ -526,6 +534,18 @@ static int mfill_atomic_pte_copy(struct mfill_state *state) {const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); +/* +* The normal page fault path for a MAP_PRIVATE mapping in a +* file-backed VMA will invoke the fault, fill the hole in the file and +* COW it right away. The result generates plain anonymous memory. +* So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll +* generate anonymous memory directly without actually filling the +* hole. For the MAP_PRIVATE case the robustness check only happens in +* the pagetable (to verify it's still none) and not in the page cache. +*/ +if (!(state->vma->vm_flags &VM_SHARED )) + ops = &anon_uffd_ops; +return __mfill_atomic_pte (state, ops); }@@ -545,7 +565,8 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)spinlock_t *ptl;int ret; -if (mm_forbids_zeropage (dst_vma->vm_mm)) +if (mm_forbids_zeropage (dst_vma->vm_mm) || + (dst_vma->vm_flags &VM_SHARED ))return mfill_atomic_pte_zeroed_folio (state); _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr),@@ -600,11 +621,10 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) } ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - page, false, flags); + page, flags);if (ret) goto out_release; -folio_unlock (folio);return 0 ; out_release:@@ -827,41 +847,18 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma; -unsigned long src_addr = state->src_addr; -unsigned long dst_addr = state->dst_addr; -struct folio **foliop = &state->folio;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd; -ssize_t err;if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ))return mfill_atomic_pte_continue (state);if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON ))return mfill_atomic_pte_poison (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) +return mfill_atomic_pte_copy (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) +return mfill_atomic_pte_zeropage (state); -/* -* The normal page fault path for a shmem will invoke the -* fault, fill the hole in the file and COW it right away. The -* result generates plain anonymous memory. So when we are -* asked to fill an hole in a MAP_PRIVATE shmem mapping, we'll -* generate anonymous memory directly without actually filling -* the hole. For the MAP_PRIVATE case the robustness check -* only happens in the pagetable (to verify it's still none) -* and not in the radix tree. -*/
mm/userfaultfd.c extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 862
Quoted 10 lines
mm/userfaultfd.c +0
-10-if (!(dst_vma->vm_flags &VM_SHARED )) { -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (state); -else - err =mfill_atomic_pte_zeropage (state); - }else { - err =shmem_mfill_atomic_pte (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); - }
It's great to merge these otherwise.
Thanks!
Quoted 9 lines
mm/userfaultfd.c +1
-2- -return err; +return -EOPNOTSUPP ; } static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,Signature · 1 line
2.51.0
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 10/17] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 26 lines · Peter Xu
Quoted 9 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Add
filemap_add()andfilemap_remove()methods tovm_uffd_opsand use them in__mfill_atomic_pte()to add shmem folios to page cache and remove them in case of error.Implement these methods in shmem along with
vm_uffd_ops->alloc_folio()and dropshmem_mfill_atomic_pte().Quoted 7 lines
Since userfaultfd now does not reference any functions from shmem, drop include if
linux/shmem_fs.hfrommm/userfaultfd.c
mfill_atomic_install_pte()is not used anywhere outside of mm/userfaultfd, make it static.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>This patch looks like a real nice cleanup on its own, thanks Mike!
I guess I never tried to read into shmem accountings, now after I read some of the codes I don't see any issue with your change. We can also wait for some shmem developers double check those. Comments inline below on something I spot.
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
fixup
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>[unexpected lines can be removed here]
Sure :)
Quoted 7 lines
Quoted 7 lines
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 20 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 155 deletions(-)
Quoted 14 lines
include/linux/shmem_fs.h struct vm_uffd_ops { · line 103
Quoted 12 lines
include/linux/shmem_fs.h +4
-0@@ -97,6 +97,21 @@ struct vm_uffd_ops {*/ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache. +* Returns 0 on success, error code on failre. failure
Thanks, will fix.
Quoted 14 lines
include/linux/shmem_fs.h struct vm_uffd_ops { · line 112
Quoted 9 lines
include/linux/shmem_fs.h +9
-0+*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +* The folio should be unlocked, but the reference to it should not be +* dropped. Might be slightly misleading to explicitly mention this? As page cache also holds references and IIUC they need to be dropped there. But I get your point, on keeping the last refcount due to allocation.
Quoted 2 lines
IMHO the "should revert the operation of ->filemap_add()" is good enough and accurately describes it.
Yeah, sounds good.
Quoted 5 lines
Quoted 5 lines
include/linux/shmem_fs.h +2
-0+*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */
...
Quoted 54 lines
Quoted 24 lines
include/linux/shmem_fs.h +10
-11+static int shmem_mfill_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +gfp_t gfp =mapping_gfp_mask (mapping); +int err; +__folio_set_locked (folio);__folio_set_swapbacked (folio); -__folio_mark_uptodate (folio); - - ret = -EFAULT ; - max_off =DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ); -if (unlikely (pgoff >= max_off)) - goto out_release; - ret =mem_cgroup_charge (folio, dst_vma->vm_mm, gfp); -if (ret) - goto out_release; - ret =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); -if (ret)include/linux/shmem_fs.h
Quoted 10 lines
include/linux/shmem_fs.h +4
-5- goto out_release; + err =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); +if (err) + goto err_unlock; - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true, flags); -if (ret) - goto out_delete_from_cache; +if (shmem_inode_acct_blocks (inode,1 )) {We used to do this early before allocation, IOW, I think we still have an option to leave this to
alloc_folio()hook. However I don't see an issue either keeping it infilemap_add(). Maybe this movement should better be spelled out in the commit message anyway on how this decision is made.IIUC it's indeed safe we move this
acct_blocks()here, I even see Hugh mentioned such in an older commit3022fd7af96, but Hugh left uffd alone at that time:Userfaultfd is a foreign country: they do things differently there, and for good reason - to avoid
mmap_lockdeadlock. Leave ordering inshmem_mfill_atomic_pte()untouched for now, but I would rather like to mesh it better withshmem_get_folio_gfp()in the future.I'm not sure if that's also what you wanted to do - to make userfaultfd code work similarly like what
shmem_alloc_and_add_folio()does right now. Maybe you want to mention that too somewhere in the commit log when posting a formal patch.
Quoted 6 lines
One thing not directly relevant is,
shmem_alloc_and_add_folio()also does proper recalc of inode allocation info whenacct_blocks()fails here. But if that's a problem, that's pre-existing for userfaultfd, so IIUC we can also leave it alone until someone (maybe quota user) complains about shmem allocation failures onUFFDIO_COPY.. It's just that it looks similar problem here in userfaultfd path.
I actually wanted to have ordering as close as possible to shmem_alloc_and_add_folio(), that's the first reason on moving acct_blocks to ->filemap_add().
Another reason, is that it simplifies rollback in case of a failure, as shmem_recalc_inode(inode, 0, 0); in ->filemap_remove() takes care of the block accounting as well.
Quoted 11 lines
include/linux/shmem_fs.h
Quoted 5 lines
include/linux/shmem_fs.h +4
-0+ err = -ENOMEM ; + goto err_delete_from_cache; + } +folio_add_lru (folio);This change is pretty separate from the work, but looks correct to me: IIUC we moved the lru add earlier now, and it should be safe as long as we're holding folio lock all through the process, and
folio_put()(ultimately,__page_cache_release()) will always properly undo the lru change. Please help double check if my understanding is correct.
This follows shmem_alloc_and_add_folio(), and my understanding as well that this is safe as long as we hold folio lock.
Quoted 5 lines
Quoted 5 lines
include/linux/shmem_fs.h +5
-0+static void shmem_mfill_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); +
Quoted 8 lines
include/linux/shmem_fs.h
Quoted 6 lines
include/linux/shmem_fs.h +2
-3+filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 );This looks wrong, or maybe I miss somewhere we did the
unacct_blocks()?
This is handled by shmem_recalc_inode(inode, 0, 0).
Quoted 9 lines
include/linux/shmem_fs.h int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 401
Quoted 6 lines
include/linux/shmem_fs.h +2
-0@@ -401,6 +397,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte); +if (page_in_cache) +folio_unlock (folio);Nitpick: another small change that looks correct, but IMHO would be nice to either make it a small separate patch, or mention in the commit message.
I'll address this in the commit log,
Quoted 4 lines
Quoted 4 lines
include/linux/shmem_fs.h +1
-0+/* No need to invalidate - it was non-present before */ update_mmu_cache (dst_vma, dst_addr, dst_pte); ret =0 ;
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH RFC 10/17] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 14 lines · Mike Rapoport
Quoted 5 lines
Quoted 5 lines
mm/shmem.c +5
-0+static void shmem_mfill_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); +Quoted 8 lines
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3282
Quoted 6 lines
mm/shmem.c +2
-3+filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 );This looks wrong, or maybe I miss somewhere we did the
unacct_blocks()?
Quoted 1 line
This is handled by
shmem_recalc_inode(inode, 0, 0).
IIUC shmem_recalc_inode() only does the fixup of shmem_inode_info over possiblly changing inode->i_mapping->nrpages. It's not for reverting the accounting in the failure paths here.
OTOH, we still need to maintain accounting for the rest things with correctly invoke shmem_inode_unacct_blocks(). One thing we can try is testing this series against either shmem quota support (since 2023, IIUC it's relevant to "quota" mount option), or max_blocks accountings (IIUC, "size" mount option), etc. Any of those should reflect a difference if my understanding is correct.
So IIUC we still need the unacct_blocks(), please kindly help double check.
Thanks,
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRe: [PATCH RFC 10/17] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 21 lines · Peter Xu
Quoted 14 lines · Mike Rapoport
Quoted 5 lines
Quoted 5 lines
mm/shmem.c +5
-0+static void shmem_mfill_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); +Quoted 8 lines
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3282
Quoted 6 lines
mm/shmem.c +2
-3+filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 );This looks wrong, or maybe I miss somewhere we did the
unacct_blocks()?Quoted 1 line
This is handled by
shmem_recalc_inode(inode, 0, 0).IIUC
shmem_recalc_inode()only does the fixup ofshmem_inode_infoover possiblly changinginode->i_mapping->nrpages. It's not for reverting the accounting in the failure paths here.
Quoted 8 lines
OTOH, we still need to maintain accounting for the rest things with correctly invoke
shmem_inode_unacct_blocks(). One thing we can try is testing this series against either shmem quota support (since 2023, IIUC it's relevant to "quota" mount option), ormax_blocksaccountings (IIUC, "size" mount option), etc. Any of those should reflect a difference if my understanding is correct.So IIUC we still need the
unacct_blocks(), please kindly help double check.
I followed shmem_get_folio_gfp() error handling, and unless I missed something we should have the same sequence with uffd.
In shmem_mfill_filemap_add() we increment both i_mapping->nrpages and info->alloced in shmem_add_to_page_cache() and shmem_recalc_inode(inode, 1, 0) respectively.
Then in shmem_filemap_remove() the call to filemap_remove_folio() decrements i_mapping->nrpages and shmem_recalc_inode(inode, 0, 0) will see freed=1 and will call shmem_inode_unacct_blocks().
Quoted 5 lines
Thanks,
Signature · 1 line
Peter Xu
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH RFC 10/17] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 34 lines · Mike Rapoport
Quoted 21 lines · Peter Xu
Quoted 14 lines · Mike Rapoport
Quoted 5 lines
Quoted 5 lines
mm/shmem.c +5
-0+static void shmem_mfill_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); +Quoted 8 lines
mm/shmem.c static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · line 3282
Quoted 6 lines
mm/shmem.c +2
-3+filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 );This looks wrong, or maybe I miss somewhere we did the
unacct_blocks()?Quoted 1 line
This is handled by
shmem_recalc_inode(inode, 0, 0).IIUC
shmem_recalc_inode()only does the fixup ofshmem_inode_infoover possiblly changinginode->i_mapping->nrpages. It's not for reverting the accounting in the failure paths here.Quoted 8 lines
OTOH, we still need to maintain accounting for the rest things with correctly invoke
shmem_inode_unacct_blocks(). One thing we can try is testing this series against either shmem quota support (since 2023, IIUC it's relevant to "quota" mount option), ormax_blocksaccountings (IIUC, "size" mount option), etc. Any of those should reflect a difference if my understanding is correct.So IIUC we still need the
unacct_blocks(), please kindly help double check.I followed
shmem_get_folio_gfp()error handling, and unless I missed something we should have the same sequence with uffd.
Quoted 7 lines
In
shmem_mfill_filemap_add()we increment bothi_mapping->nrpagesandinfo->allocedinshmem_add_to_page_cache()andshmem_recalc_inode(inode, 1, 0)respectively.Then in
shmem_filemap_remove()the call tofilemap_remove_folio()decrementsi_mapping->nrpagesandshmem_recalc_inode(inode, 0, 0)will see freed=1 and will callshmem_inode_unacct_blocks().
You're correct. I guess I was misleaded by the comments above shmem_recalc_inode() when reading this part assuming it's only for the cases where nrpages changed behind the hood.. :)
I believe we need shmem_recalc_inode(inode, 0, 0) to make sure info->alloced is properly decremented, so shmem_inode_unacct_blocks() explicit calls will miss that otherwise due to the reordering of shmem accounting in this patch.
It's slightly tricky on using these functions, I wonder if we want to mention them in the commit log, but I'm OK either way.
Thanks for double checking!
Signature · 1 line
Peter Xu
Mike Rapoport
maintainerRFC PATCH 11/17 userfaultfd: mfill_atomic() remove retry logic
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().
Drop the retry logic from mfill_atomic().
--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
@@ -29,7 +29,6 @@ struct mfill_state {struct vm_area_struct *vma;unsigned long src_addr;unsigned long dst_addr; -struct folio *folio;pmd_t *pmd; };
@@ -889,7 +888,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); -retry: err =mfill_get_vma (&state);if (err) goto out;
@@ -916,26 +914,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =mfill_atomic_pte (&state);cond_resched (); -if (unlikely (err == -ENOENT )) { -void *kaddr; - -mfill_put_vma (&state); -VM_WARN_ON_ONCE (!state.folio); - - kaddr =kmap_local_folio (state.folio,0 ); - err =copy_from_user (kaddr, - (const void __user *)state.src_addr, -PAGE_SIZE ); -kunmap_local (kaddr); -if (unlikely (err)) { - err = -EFAULT ; - goto out; - } -flush_dcache_folio (state.folio); - goto retry; - }else -VM_WARN_ON_ONCE (state.folio); -if (!err) { state.dst_addr +=PAGE_SIZE ; state.src_addr +=PAGE_SIZE ;
@@ -951,8 +929,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (state.vma)mfill_put_vma (&state); out: -if (state.folio) -folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 12/17 mm: introduce VM_FAULT_UFFD_MINOR fault reason
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When a VMA is registered with userfaulfd in minor mode, its ->fault() method should check if a folio exists in the page cache and if yes ->fault() should call handle_userfault(VM_UFFD_MINOR).
Instead of calling handle_userfault() directly from a specific ->fault() implementation introduce new fault reason VM_FAULT_UFFD_MINOR that will notify the core page fault handler that it should call handle_userfaultfd(VM_UFFD_MINOR) to complete a page fault.
Replace a call to handle_userfault(VM_UFFD_MINOR) in shmem and use the new VM_FAULT_UFFD_MINOR there instead.
For configurations that don't enable CONFIG_USERFAULTFD, VM_FAULT_UFFD_MINOR is set to 0.
--- include/linux/mm_types.h | 10 +++++++++- mm/memory.c | 5 ++++- mm/shmem.c | 2 +- 3 files changed, 14 insertions(+), 3 deletions(-)
include/linux/mm_types.h +9 -1
@@ -1555,6 +1555,8 @@ typedef __bitwise unsigned int vm_fault_t;* fsync() to complete (for synchronous page faults * in DAX) * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released +* @VM_FAULT_UFFD_MINOR: ->fault did not modify page tables and needs +* handle_userfault(VM_UFFD_MINOR) to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -1572,6 +1574,11 @@ enum vm_fault_reason {VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 ,VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , +#ifdef CONFIG_USERFAULTFD +VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x008000 , +#else +VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x000000 , +#endif VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
@@ -1596,7 +1603,8 @@ enum vm_fault_reason { { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ - { VM_FAULT_COMPLETED, "COMPLETED" } + { VM_FAULT_COMPLETED, "COMPLETED" }, \ + { VM_FAULT_UFFD_MINOR, "UFFD_MINOR" } struct vm_special_mapping {const char *name;/* The name, e.g. "[vdso]". */
mm/memory.c +4 -1
@@ -5319,8 +5319,11 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY | -VM_FAULT_DONE_COW ))) +VM_FAULT_DONE_COW |VM_FAULT_UFFD_MINOR ))) { +if (ret &VM_FAULT_UFFD_MINOR ) +return handle_userfault (vmf,VM_UFFD_MINOR );return ret; + } folio =page_folio (vmf->page);if (unlikely (PageHWPoison (vmf->page))) {
mm/shmem.c +1 -1
@@ -2481,7 +2481,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,if (folio && vma &&userfaultfd_minor (vma)) {if (!xa_is_value (folio))folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); + *fault_type =VM_FAULT_UFFD_MINOR ;return 0 ; }
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 13/17 mm: introduce VM_FAULT_UFFD_MISSING fault reason
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
When a VMA is registered with userfaulfd in missing mode, its ->fault() method should check if a folio exists in the page cache and if no ->fault() should call handle_userfault(VM_UFFD_MISSING).
Instead of calling handle_userfault() directly from a specific ->fault() implementation introduce new fault reason VM_FAULT_UFFD_MISSING that will notify the core page fault handler that it should call handle_userfaultfd(VM_UFFD_MISSING) to complete a page fault.
Replace a call to handle_userfault(VM_UFFD_MISSING) in shmem and use the new VM_FAULT_UFFD_MISSING there instead.
For configurations that don't enable CONFIG_USERFAULTFD, VM_FAULT_UFFD_MISSING is set to 0.
--- include/linux/mm_types.h | 7 ++++++- mm/memory.c | 5 ++++- mm/shmem.c | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-)
include/linux/mm_types.h +6 -1
@@ -1557,6 +1557,8 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released * @VM_FAULT_UFFD_MINOR: ->fault did not modify page tables and needs * handle_userfault(VM_UFFD_MINOR) to complete +* @VM_FAULT_UFFD_MISSING: ->fault did not modify page tables and needs +* handle_userfault(VM_UFFD_MISSING) to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -1576,8 +1578,10 @@ enum vm_fault_reason { VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,#ifdef CONFIG_USERFAULTFD VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x008000 , +VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x010000 , #elseVM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x000000 , +VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x000000 , #endif VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
@@ -1604,7 +1608,8 @@ enum vm_fault_reason { { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ { VM_FAULT_COMPLETED, "COMPLETED" }, \ - { VM_FAULT_UFFD_MINOR, "UFFD_MINOR" } + { VM_FAULT_UFFD_MINOR, "UFFD_MINOR" }, \ + { VM_FAULT_UFFD_MISSING, "UFFD_MISSING" } struct vm_special_mapping {const char *name;/* The name, e.g. "[vdso]". */
mm/memory.c +4 -1
@@ -5319,9 +5319,12 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY | -VM_FAULT_DONE_COW |VM_FAULT_UFFD_MINOR ))) { +VM_FAULT_DONE_COW |VM_FAULT_UFFD_MINOR | +VM_FAULT_UFFD_MISSING ))) {if (ret &VM_FAULT_UFFD_MINOR )return handle_userfault (vmf,VM_UFFD_MINOR ); +if (ret &VM_FAULT_UFFD_MISSING ) +return handle_userfault (vmf,VM_UFFD_MISSING );return ret; }
mm/shmem.c +1 -1
@@ -2530,7 +2530,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,*/ if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); + *fault_type =VM_FAULT_UFFD_MISSING ;return 0 ; }
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 14/17 KVM: guest_memfd: implement userfaultfd minor mode
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
userfaultfd notifications about minor page faults used for live migration and snapshotting of VMs with memory backed by shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").
To use the same mechanism for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd minor mode.
Extend ->fault() method of guest_memfd with ability to notify core page fault handler that a page fault requires handle_userfault(VM_UFFD_MINOR) to complete and add vm_uffd_ops to guest_memfd vm_ops with implementation of ->can_userfault() and ->get_folio_noalloc() methods.
--- virt/kvm/guest_memfd.c | 76 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 11 deletions(-)
virt/kvm/guest_memfd.c +65 -11
@@ -7,6 +7,7 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h"
@@ -121,6 +122,26 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,return r; } +static struct folio *kvm_gmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +return __filemap_get_folio (inode->i_mapping, pgoff, +FGP_LOCK |FGP_ACCESSED ,0 ); +} + +static struct folio *__kvm_gmem_folio_alloc (struct inode *inode,pgoff_t index) +{ +struct mempolicy *policy; +struct folio *folio; + + policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); + folio =__filemap_get_folio_mpol (inode->i_mapping, index, +FGP_LOCK |FGP_ACCESSED |FGP_CREAT , +mapping_gfp_mask (inode->i_mapping), policy); +mpol_cond_put (policy); + +return folio; +} +/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest.
@@ -133,25 +154,17 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,static struct folio *kvm_gmem_get_folio (struct inode *inode,pgoff_t index) {/* TODO: Support huge pages. */ -struct mempolicy *policy;struct folio *folio;/* * Fast-path: See if folio is already present in mapping to avoid * policy_lookup. */ - folio =__filemap_get_folio (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED ,0 ); + folio =kvm_gmem_get_folio_noalloc (inode, index);if (!IS_ERR (folio))return folio; - policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); - folio =__filemap_get_folio_mpol (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED |FGP_CREAT , -mapping_gfp_mask (inode->i_mapping), policy); -mpol_cond_put (policy); - -return folio; +return __kvm_gmem_folio_alloc (inode, index); }static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter (struct inode *inode)
@@ -405,7 +418,24 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED ))return VM_FAULT_SIGBUS ; - folio =kvm_gmem_get_folio (inode, vmf->pgoff); + folio =__filemap_get_folio (inode->i_mapping, vmf->pgoff, +FGP_LOCK |FGP_ACCESSED ,0 ); + +if (userfaultfd_armed (vmf->vma)) { +/* +* If userfaultfd is registered in minor mode and a folio +* exists, return VM_FAULT_UFFD_MINOR to trigger the +* userfaultfd handler. +*/ +if (userfaultfd_minor (vmf->vma) && !IS_ERR_OR_NULL (folio)) { + ret =VM_FAULT_UFFD_MINOR ; + goto out_folio; + } + } + +/* folio not in the pagecache, try to allocate */ +if (IS_ERR (folio)) + folio =__kvm_gmem_folio_alloc (inode, vmf->pgoff);if (IS_ERR (folio)) {if (PTR_ERR (folio) == -EAGAIN )return VM_FAULT_RETRY ;
@@ -462,12 +492,36 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, }#endif /* CONFIG_NUMA */ +#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +struct inode *inode =file_inode (vma->vm_file); + +/* +* Only support userfaultfd for guest_memfd with INIT_SHARED flag. +* This ensures the memory can be mapped to userspace. +*/ +if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )) +return false; + +return true; +} + +static const struct vm_uffd_ops kvm_gmem_uffd_ops = { + .can_userfault = kvm_gmem_can_userfault, + .get_folio_noalloc = kvm_gmem_get_folio_noalloc, +}; +#endif /* CONFIG_USERFAULTFD */ +static const struct vm_operations_struct kvm_gmem_vm_ops = { .fault = kvm_gmem_fault_user_mapping, #ifdef CONFIG_NUMA .get_policy = kvm_gmem_get_policy, .set_policy = kvm_gmem_set_policy, #endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &kvm_gmem_uffd_ops, +#endif };static int kvm_gmem_mmap (struct file *file,struct vm_area_struct *vma)
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 15/17 KVM: guest_memfd: implement userfaultfd missing mode
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
userfaultfd missing mode allows populating guest memory with the content supplied by userspace on demand.
Extend guest_memfd implementation of vm_uffd_ops to support MISSING mode.
--- virt/kvm/guest_memfd.c | 60 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)
virt/kvm/guest_memfd.c +59 -1
@@ -431,6 +431,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) ret =VM_FAULT_UFFD_MINOR ; goto out_folio; } + +/* +* Check if userfaultfd is registered in missing mode. If so, +* check if a folio exists in the page cache. If not, return +* VM_FAULT_UFFD_MISSING to trigger the userfaultfd handler. +*/ +if (userfaultfd_missing (vmf->vma) &&IS_ERR_OR_NULL (folio)) +return VM_FAULT_UFFD_MISSING ; }/* folio not in the pagecache, try to allocate */
@@ -507,9 +515,59 @@ static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flareturn true; } +static struct folio *kvm_gmem_folio_alloc (struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); +struct mempolicy *mpol; +struct folio *folio; +gfp_t gfp; + +if (unlikely (pgoff >= (i_size_read (inode) >>PAGE_SHIFT ))) +return NULL ; + + gfp =mapping_gfp_mask (inode->i_mapping); + mpol =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, pgoff); + mpol = mpol ?:get_task_policy (current); + folio =folio_alloc_mpol (gfp,0 , mpol, pgoff,numa_node_id ()); +mpol_cond_put (mpol); + +return folio; +} + +static int kvm_gmem_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +int err; + +__folio_set_locked (folio); + err =filemap_add_folio (mapping, folio, pgoff,GFP_KERNEL ); +if (err) { +folio_unlock (folio); +return err; + } + +return 0 ; +} + +static void kvm_gmem_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +filemap_remove_folio (folio); +folio_unlock (folio); +} +static const struct vm_uffd_ops kvm_gmem_uffd_ops = { - .can_userfault = kvm_gmem_can_userfault, + .can_userfault = kvm_gmem_can_userfault, .get_folio_noalloc = kvm_gmem_get_folio_noalloc, + .alloc_folio = kvm_gmem_folio_alloc, + .filemap_add = kvm_gmem_filemap_add, + .filemap_remove = kvm_gmem_filemap_remove, };#endif /* CONFIG_USERFAULTFD */
Signature · 1 line
2.51.0
Re: [PATCH RFC 15/17] KVM: guest_memfd: implement userfaultfd missing mode
Quoted 44 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd missing mode allows populating guest memory with the content supplied by userspace on demand.
Extend
guest_memfdimplementation ofvm_uffd_opsto support MISSING mode.Signed-off-by Nikita Kalyazin <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- virt/kvm/guest_memfd.c | 60 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)virt/kvm/guest_memfd.c +14
-0@@ -431,6 +431,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) ret =VM_FAULT_UFFD_MINOR ; goto out_folio; } + +/* +* Check if userfaultfd is registered in missing mode. If so, +* check if a folio exists in the page cache. If not, return +* VM_FAULT_UFFD_MISSING to trigger the userfaultfd handler. +*/ +if (userfaultfd_missing (vmf->vma) &&IS_ERR_OR_NULL (folio)) +return VM_FAULT_UFFD_MISSING ; }/* folio not in the pagecache, try to allocate */ @@ -507,9 +515,59 @@ static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flareturn true; } +static struct folio *kvm_gmem_folio_alloc (struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); +struct mempolicy *mpol;
virt/kvm/guest_memfd.c static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_fla · line 533
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+struct folio *folio; +gfp_t gfp; + +if (unlikely (pgoff >= (i_size_read (inode) >>PAGE_SHIFT ))) +return NULL ; + + gfp =mapping_gfp_mask (inode->i_mapping); + mpol =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, pgoff); + mpol = mpol ?:get_task_policy (current); + folio =folio_alloc_mpol (gfp,0 , mpol, pgoff,numa_node_id ());
It looks like folio_alloc_mpol_noprof() and filemap_remove_folio() are not actually exported to modules. Would it be ok to export them similar to how it was done in f634f10809ec ("mm/mempolicy: Export memory policy symbols")?
mm/filemap.c +1 -0
@@ -261,6 +261,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache
mm/mempolicy.c +1 -0
@@ -2442,6 +2442,7 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,set_page_refcounted (page);return page_rmappable_folio (page); } +EXPORT_SYMBOL_FOR_MODULES (folio_alloc_mpol_noprof,"kvm" );/** * vma_alloc_folio - Allocate a folio for a VMA.
Quoted 44 lines
virt/kvm/guest_memfd.c +35
-2+mpol_cond_put (mpol); + +return folio; +} + +static int kvm_gmem_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +int err; + +__folio_set_locked (folio); + err =filemap_add_folio (mapping, folio, pgoff,GFP_KERNEL ); +if (err) { +folio_unlock (folio); +return err; + } + +return 0 ; +} + +static void kvm_gmem_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +filemap_remove_folio (folio); +folio_unlock (folio); +} +static const struct vm_uffd_ops kvm_gmem_uffd_ops = { - .can_userfault = kvm_gmem_can_userfault, + .can_userfault = kvm_gmem_can_userfault, .get_folio_noalloc = kvm_gmem_get_folio_noalloc, + .alloc_folio = kvm_gmem_folio_alloc, + .filemap_add = kvm_gmem_filemap_add, + .filemap_remove = kvm_gmem_filemap_remove, };#endif /* CONFIG_USERFAULTFD */ --2.51.0
Re: [PATCH RFC 15/17] KVM: guest_memfd: implement userfaultfd missing mode
+ David's new email
Quoted 139 lines · Nikita Kalyazin
Quoted 50 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd missing mode allows populating guest memory with the content supplied by userspace on demand.
Extend
guest_memfdimplementation ofvm_uffd_opsto support MISSING mode.Signed-off-by Nikita Kalyazin <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>---
virt/kvm/guest_memfd.c| 60 +++++++++++++++++++++++++++++++++++++++++-1 file changed, 59 insertions(+), 1 deletion(-)virt/kvm/guest_memfd.c +4
-0@@ -431,6 +431,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) ret = VM_FAULT_UFFD_MINOR; goto out_folio; } + + /* +* Check if userfaultfd is registered in missing mode. If so, +* check if a folio exists in the page cache. If not, return
+ *VM_FAULT_UFFD_MISSINGto trigger the userfaultfd handler.
+ */
+ if (userfaultfd_missing(vmf->vma)&&IS_ERR_OR_NULL(folio))
+ returnVM_FAULT_UFFD_MISSING;
}/* folio not in the pagecache, try to allocate */
virt/kvm/guest_memfd.c +0
-0@@ -507,9 +515,59 @@ static bool kvm_gmem_can_userfault(struct
vm_area_struct*vma,vm_flags_tvm_fla
return true;
}virt/kvm/guest_memfd.c +6
-0+static struct folio *kvm_gmem_folio_alloc (struct vm_area_struct *vma, + unsigned long addr) +{ + struct inode *inode = file_inode(vma->vm_file); + pgoff_t pgoff = linear_page_index(vma, addr); + struct mempolicy *mpol;virt/kvm/guest_memfd.c
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ struct folio *folio; + gfp_t gfp; + + if (unlikely(pgoff >= (i_size_read(inode) >> PAGE_SHIFT))) + return NULL; + + gfp = mapping_gfp_mask(inode->i_mapping); + mpol = mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff); + mpol = mpol ?: get_task_policy(current); + folio = folio_alloc_mpol(gfp, 0, mpol, pgoff, numa_node_id());It looks like
folio_alloc_mpol_noprof()andfilemap_remove_folio()are not actually exported to modules. Would it be ok to export them similar to how it was done inf634f10809ec("mm/mempolicy: Export memory policy symbols")?mm/filemap.c +1
-0@@ -261,6 +261,7 @@ void filemap_remove_folio(struct folio *folio) filemap_free_folio(mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" ); /** page_cache_delete_batch - delete several folios from page cache mm/mempolicy.c +1
-0@@ -2442,6 +2442,7 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp,unsigned int order, set_page_refcounted(page); return page_rmappable_folio(page); } +EXPORT_SYMBOL_FOR_MODULES (folio_alloc_mpol_noprof,"kvm" ); /***
vma_alloc_folio- Allocate a folio for a VMA.[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=f634f10809ec3d51d9529dfb0f99bc7cec1b5eff
Quoted 44 lines
+
mpol_cond_put(mpol);virt/kvm/guest_memfd.c +30
-0+ + return folio; +} + +static int kvm_gmem_filemap_add (struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr) +{ + struct inode *inode = file_inode(vma->vm_file); + struct address_space *mapping = inode->i_mapping; + pgoff_t pgoff = linear_page_index(vma, addr); + int err; + + __folio_set_locked(folio); + err = filemap_add_folio(mapping, folio, pgoff, GFP_KERNEL); + if (err) { + folio_unlock(folio); + return err; + } + + return 0; +} + +static void kvm_gmem_filemap_remove (struct folio *folio, + struct vm_area_struct *vma) +{ + filemap_remove_folio(folio); + folio_unlock(folio); +} +static const struct
vm_uffd_opskvm_gmem_uffd_ops= { - .can_userfault =kvm_gmem_can_userfault, + .can_userfault =kvm_gmem_can_userfault, .get_folio_noalloc =kvm_gmem_get_folio_noalloc, + .alloc_folio =kvm_gmem_folio_alloc, + .filemap_add =kvm_gmem_filemap_add, + .filemap_remove =kvm_gmem_filemap_remove, };
#endif /*CONFIG_USERFAULTFD*/Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 16/17 KVM: selftests: test userfaultfd minor for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
@@ -10,13 +10,17 @@#include <errno.h> #include <stdio.h> #include <fcntl.h> +#include <pthread.h> #include <linux/bitmap.h> #include <linux/falloc.h> #include <linux/sizes.h> +#include <linux/userfaultfd.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> #include "kvm_util.h" #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)close (fd1); } +struct fault_args { +char *addr; +char value; +}; + +static void *fault_thread_fn (void *arg) +{ +struct fault_args *args = arg; + +/* Trigger page fault */ + args->value = *args->addr; +return NULL ; +} + +static void test_uffd_minor (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_continue uffd_cont; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *mem_nofault, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xcd ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + +/* Map the guest_memfd twice: once with UFFD registered, once without */ + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + mem_nofault =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem_nofault !=MAP_FAILED ,"mmap should succeed" ); + +/* Register UFFD_MINOR on the first mapping */ + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MINOR ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + +/* +* Populate the page in the page cache first via mem_nofault. +* This is required for UFFD_MINOR - the page must exist in the cache. +* Write test data to the page. +*/ +memcpy (mem_nofault + offset, buf + offset, page_size); + +/* +* Now access the same page via mem (which has UFFD_MINOR registered). +* Since the page exists in the cache, this should trigger UFFD_MINOR. +*/ + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_MINOR , +"pagefault should be minor fault" ); + +/* Resolve the minor fault with UFFDIO_CONTINUE */ + uffd_cont.range.start = (unsigned long )fault_addr; + uffd_cont.range.len = page_size; + uffd_cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &uffd_cont); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_CONTINUE) should succeed" ); + +/* Wait for the faulting thread to complete */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* Verify the thread read the correct value */ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was written" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem_nofault, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (file_size, vm, flags);gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); + +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +gmem_test (uffd_minor, vm, flags); }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRFC PATCH 17/17 KVM: selftests: test userfaultfd missing for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)close (uffd); } +static void test_uffd_missing (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_copy uffd_copy; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xab ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MISSING ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (!(msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ), +"pagefault should not be write-protect" ); + + uffd_copy.dst = (unsigned long )fault_addr; + uffd_copy.src = (unsigned long )(buf + offset); + uffd_copy.len = page_size; + uffd_copy.mode =0 ; + ret =ioctl (uffd,UFFDIO_COPY , &uffd_copy); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_COPY) should succeed" ); + +/* Wait for the faulting thread to complete - this provides the memory barrier */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* +* Now it's safe to check args.value - the thread has completed +* and memory is synchronized +*/ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was copied" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); -if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) {gmem_test (uffd_minor, vm, flags); +gmem_test (uffd_missing, vm, flags); + } }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.51.0
Peter Xu
reviewerQuoted 56 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in
guest_memfd. They are quite different from the latest posting [1] so I'm restarting the versioning. As there was a lot of tension around the topic, this is an RFC to get some feedback and see how we can move forward.As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use
vm_uffd_opsfor allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed inUFFDIO_COPYwas copied and remove the folio from the page cache if page table update fails.In order for
guest_memfdto notify userspace about page faults, there are newVM_FAULT_UFFD_MINORandVM_FAULT_UFFD_MISSINGthat a ->fault() handler can return to inform the page fault handler that it needs to callhandle_userfault()to complete the fault.Nikita helped to plumb these new goodies into
guest_memfdand provided basic tests to verify thatguest_memfdworks with userfaultfd.I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what
vm_uffd_opsprovides and people objected to exposing such low level APIs as a part of VMA operations.Also, to enable uffd in
guest_memfdrefactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.[1] https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (12):
userfaultfd: introducemfill_copy_folio_locked()helper
userfaultfd: introduce structmfill_state
userfaultfd: introducemfill_get_pmd()helper.
userfaultfd: introducemfill_get_vma()andmfill_put_vma()
userfaultfd: retry copying with locks dropped inmfill_atomic_pte_copy()
userfaultfd: movevma_can_userfaultout of line
userfaultfd: introducevm_uffd_ops
userfaultfd, shmem: use a VMA callback to handleUFFDIO_CONTINUE
userfaultfd: introducevm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations usingvm_uffd_ops
userfaultfd:mfill_atomic()remove retry logic
mm: introduceVM_FAULT_UFFD_MINORfault reasonNikita Kalyazin (5):
mm: introduceVM_FAULT_UFFD_MISSINGfault reason
KVM:guest_memfd: implement userfaultfd minor mode
KVM:guest_memfd: implement userfaultfd missing mode
KVM: selftests: test userfaultfd minor forguest_memfd
KVM: selftests: test userfaultfd missing forguest_memfd
Quoted 11 lines
include/linux/mm.h | 5 + include/linux/mm_types.h | 15 +- include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 74 +- mm/hugetlb.c | 21 + mm/memory.c | 8 +- mm/shmem.c | 188 +++-- mm/userfaultfd.c | 671 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 134 +++- 10 files changed, 871 insertions(+), 450 deletions(-)
Mike,
The idea looks good to me, thanks for this work! Your process on UFFDIO_COPY over anon/shmem is nice to me.
If you remember, I used to raise a concern on introducing two new fault retvals only for userfaultfd:
https://lore.kernel.org/all/[email protected]/
IMHO they're not only unnecessarily leaking userfaultfd information into fault core definitions, but also cause code duplications. I still think we should avoid them.
This time, I've attached a smoke tested patch removing both of them.
It's pretty small and it runs all fine with all old/new userfaultfd tests (including gmem ones). Feel free to have a look at the end.
I understand you want to avoid adding mnore complexity to this series, if you want I can also prepare such a patch after this series landed to remove the two retvals. I'd still would like to know how you think about it, though, let me know if you have any comments.
Note that it may indeed need some perf tests to make sure there's zero overhead after this change. Currently there's still some trivial overheads (e.g. unnecessary folio locks), but IIUC we can even avoid that.
Thanks,
===8<===
From 5379d084494b17281f3e5365104a7edbdbe53759 Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Tue, 3 Feb 2026 15:07:58 -0500
Subject: [PATCH] mm/userfaultfd: Remove two userfaultfd fault retvals
They're not needed when with vm_uffd_ops. We can remove both of them. Actually, another side benefit is drivers do not need to process userfaultfd missing / minor faults anymore in the main fault handler.
This patch will make get_folio_noalloc() required for either MISSING or MINOR fault, but that's not a problem, as it should be lightweight and the current only outside-mm user (gmem) will support both anyway.
--- include/linux/mm_types.h | 15 +----------- include/linux/userfaultfd_k.h | 2 +- mm/memory.c | 45 +++++++++++++++++++++++++++++------ mm/shmem.c | 12 ---------- virt/kvm/guest_memfd.c | 20 ---------------- 5 files changed, 40 insertions(+), 54 deletions(-)
include/linux/mm_types.h +1 -14
@@ -1612,10 +1612,6 @@ typedef __bitwise unsigned int vm_fault_t;* fsync() to complete (for synchronous page faults * in DAX) * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released -* @VM_FAULT_UFFD_MINOR: ->fault did not modify page tables and needs -* handle_userfault(VM_UFFD_MINOR) to complete -* @VM_FAULT_UFFD_MISSING: ->fault did not modify page tables and needs -* handle_userfault(VM_UFFD_MISSING) to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -1633,13 +1629,6 @@ enum vm_fault_reason {VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 ,VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , -#ifdef CONFIG_USERFAULTFD -VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x008000 , -VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x010000 , -#else -VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x000000 , -VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x000000 , -#endif VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
@@ -1664,9 +1653,7 @@ enum vm_fault_reason { { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ - { VM_FAULT_COMPLETED, "COMPLETED" }, \ - { VM_FAULT_UFFD_MINOR, "UFFD_MINOR" }, \ - { VM_FAULT_UFFD_MISSING, "UFFD_MISSING" } + { VM_FAULT_COMPLETED, "COMPLETED" }struct vm_special_mapping {const char *name;/* The name, e.g. "[vdso]". */
include/linux/userfaultfd_k.h +1 -1
@@ -85,7 +85,7 @@ struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags);/* -* Called to resolve UFFDIO_CONTINUE request. +* Required by any uffd driver for either MISSING or MINOR fault. * Should return the folio found at pgoff in the VMA's pagecache if it * exists or ERR_PTR otherwise. * The returned folio is locked and with reference held.
mm/memory.c +38 -7
60 lines not on this page · raw message
mm/shmem.c +0 -12
@@ -2484,13 +2484,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =VM_FAULT_UFFD_MINOR ; -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);
@@ -2535,11 +2528,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =VM_FAULT_UFFD_MISSING ; -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {
virt/kvm/guest_memfd.c +0 -20
@@ -421,26 +421,6 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) folio =__filemap_get_folio (inode->i_mapping, vmf->pgoff,FGP_LOCK |FGP_ACCESSED ,0 ); -if (userfaultfd_armed (vmf->vma)) { -/* -* If userfaultfd is registered in minor mode and a folio -* exists, return VM_FAULT_UFFD_MINOR to trigger the -* userfaultfd handler. -*/ -if (userfaultfd_minor (vmf->vma) && !IS_ERR_OR_NULL (folio)) { - ret =VM_FAULT_UFFD_MINOR ; - goto out_folio; - } - -/* -* Check if userfaultfd is registered in missing mode. If so, -* check if a folio exists in the page cache. If not, return -* VM_FAULT_UFFD_MISSING to trigger the userfaultfd handler. -*/ -if (userfaultfd_missing (vmf->vma) &&IS_ERR_OR_NULL (folio)) -return VM_FAULT_UFFD_MISSING ; - } -/* folio not in the pagecache, try to allocate */ if (IS_ERR (folio)) folio =__kvm_gmem_folio_alloc (inode, vmf->pgoff);
Signature · 5 lines
2.50.1 -- Peter Xu
David Hildenbrand (Arm)
maintainerQuoted 292 lines · Peter Xu
Quoted 56 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in
guest_memfd. They are quite different from the latest posting [1] so I'm restarting the versioning. As there was a lot of tension around the topic, this is an RFC to get some feedback and see how we can move forward.As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use
vm_uffd_opsfor allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed inUFFDIO_COPYwas copied and remove the folio from the page cache if page table update fails.In order for
guest_memfdto notify userspace about page faults, there are newVM_FAULT_UFFD_MINORandVM_FAULT_UFFD_MISSINGthat a ->fault() handler can return to inform the page fault handler that it needs to callhandle_userfault()to complete the fault.Nikita helped to plumb these new goodies into
guest_memfdand provided basic tests to verify thatguest_memfdworks with userfaultfd.I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what
vm_uffd_opsprovides and people objected to exposing such low level APIs as a part of VMA operations.Also, to enable uffd in
guest_memfdrefactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.[1] https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (12):
userfaultfd: introducemfill_copy_folio_locked()helper
userfaultfd: introduce structmfill_state
userfaultfd: introducemfill_get_pmd()helper.
userfaultfd: introducemfill_get_vma()andmfill_put_vma()
userfaultfd: retry copying with locks dropped inmfill_atomic_pte_copy()
userfaultfd: movevma_can_userfaultout of line
userfaultfd: introducevm_uffd_ops
userfaultfd, shmem: use a VMA callback to handleUFFDIO_CONTINUE
userfaultfd: introducevm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations usingvm_uffd_ops
userfaultfd:mfill_atomic()remove retry logic
mm: introduceVM_FAULT_UFFD_MINORfault reasonNikita Kalyazin (5):
mm: introduceVM_FAULT_UFFD_MISSINGfault reason
KVM:guest_memfd: implement userfaultfd minor mode
KVM:guest_memfd: implement userfaultfd missing mode
KVM: selftests: test userfaultfd minor forguest_memfd
KVM: selftests: test userfaultfd missing forguest_memfdQuoted 11 lines
include/linux/mm.h | 5 + include/linux/mm_types.h | 15 +- include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 74 +- mm/hugetlb.c | 21 + mm/memory.c | 8 +- mm/shmem.c | 188 +++-- mm/userfaultfd.c | 671 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 134 +++- 10 files changed, 871 insertions(+), 450 deletions(-)Mike,
The idea looks good to me, thanks for this work! Your process on
UFFDIO_COPYover anon/shmem is nice to me.If you remember, I used to raise a concern on introducing two new fault retvals only for userfaultfd:
https://lore.kernel.org/all/[email protected]/
IMHO they're not only unnecessarily leaking userfaultfd information into fault core definitions, but also cause code duplications. I still think we should avoid them.
This time, I've attached a smoke tested patch removing both of them.
It's pretty small and it runs all fine with all old/new userfaultfd tests (including gmem ones). Feel free to have a look at the end.
I understand you want to avoid adding mnore complexity to this series, if you want I can also prepare such a patch after this series landed to remove the two retvals. I'd still would like to know how you think about it, though, let me know if you have any comments.
Note that it may indeed need some perf tests to make sure there's zero overhead after this change. Currently there's still some trivial overheads (e.g. unnecessary folio locks), but IIUC we can even avoid that.
Thanks,
===8<===
From5379d084494b17281f3e5365104a7edbdbe53759Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Tue, 3 Feb 2026 15:07:58 -0500
Subject: [PATCH] mm/userfaultfd: Remove two userfaultfd fault retvalsThey're not needed when with
vm_uffd_ops. We can remove both of them. Actually, another side benefit is drivers do not need to process userfaultfd missing / minor faults anymore in the main fault handler.This patch will make
get_folio_noalloc()required for either MISSING or MINOR fault, but that's not a problem, as it should be lightweight and the current only outside-mm user (gmem) will support both anyway.Signed-off-by Peter Xu <[email protected]>--- include/linux/mm_types.h | 15 +----------- include/linux/userfaultfd_k.h | 2 +- mm/memory.c | 45 +++++++++++++++++++++++++++++------ mm/shmem.c | 12 ---------- virt/kvm/guest_memfd.c | 20 ---------------- 5 files changed, 40 insertions(+), 54 deletions(-)include/linux/mm_types.h +1
-14@@ -1612,10 +1612,6 @@ typedef __bitwise unsigned int vm_fault_t;* fsync() to complete (for synchronous page faults * in DAX) * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released -* @VM_FAULT_UFFD_MINOR: ->fault did not modify page tables and needs -* handle_userfault(VM_UFFD_MINOR) to complete -* @VM_FAULT_UFFD_MISSING: ->fault did not modify page tables and needs -* handle_userfault(VM_UFFD_MISSING) to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */ @@ -1633,13 +1629,6 @@ enum vm_fault_reason {VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 ,VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , -#ifdef CONFIG_USERFAULTFD -VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x008000 , -VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x010000 , -#else -VM_FAULT_UFFD_MINOR = (__force vm_fault_t )0x000000 , -VM_FAULT_UFFD_MISSING = (__force vm_fault_t )0x000000 , -#endif VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };@@ -1664,9 +1653,7 @@ enum vm_fault_reason { { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ - { VM_FAULT_COMPLETED, "COMPLETED" }, \ - { VM_FAULT_UFFD_MINOR, "UFFD_MINOR" }, \ - { VM_FAULT_UFFD_MISSING, "UFFD_MISSING" } + { VM_FAULT_COMPLETED, "COMPLETED" }struct vm_special_mapping {const char *name;/* The name, e.g. "[vdso]". */ include/linux/userfaultfd_k.h +1
-1@@ -85,7 +85,7 @@ struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags);/* -* Called to resolve UFFDIO_CONTINUE request. +* Required by any uffd driver for either MISSING or MINOR fault. * Should return the folio found at pgoff in the VMA's pagecache if it * exists or ERR_PTR otherwise. * The returned folio is locked and with reference held. mm/memory.c +38
-7@@ -5338,6 +5338,33 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +static vm_fault_t fault_process_userfaultfd (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode =file_inode (vma->vm_file); +/* +* NOTE: we could double check this hook present when +* UFFDIO_REGISTER on MISSING or MINOR for a file driver. +*/ +struct folio *folio = + vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff); + +if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ +folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) +return handle_userfault (vmf,VM_UFFD_MINOR ); + }else { +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value. @@ -5370,16 +5397,20 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/* +* If this is an userfaultfd trap, process it in advance before +* triggering the genuine fault handler. +*/ +if (userfaultfd_missing (vma) ||userfaultfd_minor (vma)) { + ret =fault_process_userfaultfd (vmf); +if (ret) +return ret; + } + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY | -VM_FAULT_DONE_COW |VM_FAULT_UFFD_MINOR | -VM_FAULT_UFFD_MISSING ))) { -if (ret &VM_FAULT_UFFD_MINOR ) -return handle_userfault (vmf,VM_UFFD_MINOR ); -if (ret &VM_FAULT_UFFD_MISSING ) -return handle_userfault (vmf,VM_UFFD_MISSING ); +VM_FAULT_DONE_COW )))return ret; - } folio =page_folio (vmf->page);if (unlikely (PageHWPoison (vmf->page))) {mm/shmem.c +0
-12@@ -2484,13 +2484,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =VM_FAULT_UFFD_MINOR ; -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);@@ -2535,11 +2528,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =VM_FAULT_UFFD_MISSING ; -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {virt/kvm/guest_memfd.c +0
-13@@ -421,26 +421,6 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) folio =__filemap_get_folio (inode->i_mapping, vmf->pgoff,FGP_LOCK |FGP_ACCESSED ,0 ); -if (userfaultfd_armed (vmf->vma)) { -/* -* If userfaultfd is registered in minor mode and a folio -* exists, return VM_FAULT_UFFD_MINOR to trigger the -* userfaultfd handler. -*/ -if (userfaultfd_minor (vmf->vma) && !IS_ERR_OR_NULL (folio)) { - ret =VM_FAULT_UFFD_MINOR ; - goto out_folio; - } - -/* -* Check if userfaultfd is registered in missing mode. If so,
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) · line 426
Quoted 10 lines
virt/kvm/guest_memfd.c +0
-7-* check if a folio exists in the page cache. If not, return -* VM_FAULT_UFFD_MISSING to trigger the userfaultfd handler. -*/ -if (userfaultfd_missing (vmf->vma) &&IS_ERR_OR_NULL (folio)) -return VM_FAULT_UFFD_MISSING ; - } -/* folio not in the pagecache, try to allocate */ if (IS_ERR (folio)) folio =__kvm_gmem_folio_alloc (inode, vmf->pgoff);
That looks better in general. We should likely find a better/more consistent name for fault_process_userfaultfd().
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerQuoted 36 lines · David Hildenbrand (Arm)
Quoted 34 lines · Peter Xu
mm/memory.c +28
-0+static vm_fault_t fault_process_userfaultfd (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode =file_inode (vma->vm_file); +/* +* NOTE: we could double check this hook present when +* UFFDIO_REGISTER on MISSING or MINOR for a file driver. +*/ +struct folio *folio = + vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff); + +if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ +folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) +return handle_userfault (vmf,VM_UFFD_MINOR ); + }else { +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value. @@ -5370,16 +5397,20 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/*
Quoted 8 lines
mm/memory.c static vm_fault_t __do_fault(struct vm_fault *vmf) · line 5408
Quoted 8 lines
mm/memory.c +8
-0+* If this is an userfaultfd trap, process it in advance before +* triggering the genuine fault handler. +*/ +if (userfaultfd_missing (vma) ||userfaultfd_minor (vma)) { + ret =fault_process_userfaultfd (vmf); +if (ret) +return ret; + }
I agree this is neater than handling VM_FAULT_UFFD.
I'd just move the checks for userfaultfd_minor() and userfaultfd_missing() inside fault_process_userfaultfd().
Quoted 67 lines
Quoted 67 lines
mm/memory.c +2
-7+ ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY | -VM_FAULT_DONE_COW |VM_FAULT_UFFD_MINOR | -VM_FAULT_UFFD_MISSING ))) { -if (ret &VM_FAULT_UFFD_MINOR ) -return handle_userfault (vmf,VM_UFFD_MINOR ); -if (ret &VM_FAULT_UFFD_MISSING ) -return handle_userfault (vmf,VM_UFFD_MISSING ); +VM_FAULT_DONE_COW )))return ret; - } folio =page_folio (vmf->page);if (unlikely (PageHWPoison (vmf->page))) {mm/shmem.c +0
-12@@ -2484,13 +2484,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =VM_FAULT_UFFD_MINOR ; -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);@@ -2535,11 +2528,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =VM_FAULT_UFFD_MISSING ; -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {virt/kvm/guest_memfd.c +0
-18@@ -421,26 +421,6 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) folio =__filemap_get_folio (inode->i_mapping, vmf->pgoff,FGP_LOCK |FGP_ACCESSED ,0 ); -if (userfaultfd_armed (vmf->vma)) { -/* -* If userfaultfd is registered in minor mode and a folio -* exists, return VM_FAULT_UFFD_MINOR to trigger the -* userfaultfd handler. -*/ -if (userfaultfd_minor (vmf->vma) && !IS_ERR_OR_NULL (folio)) { - ret =VM_FAULT_UFFD_MINOR ; - goto out_folio; - } - -/* -* Check if userfaultfd is registered in missing mode. If so, -* check if a folio exists in the page cache. If not, return -* VM_FAULT_UFFD_MISSING to trigger the userfaultfd handler. -*/ -if (userfaultfd_missing (vmf->vma) &&IS_ERR_OR_NULL (folio)) -return VM_FAULT_UFFD_MISSING ;
Quoted 8 lines
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) · line 426
Quoted 5 lines
virt/kvm/guest_memfd.c +0
-2- } -/* folio not in the pagecache, try to allocate */ if (IS_ERR (folio)) folio =__kvm_gmem_folio_alloc (inode, vmf->pgoff);That looks better in general. We should likely find a better/more consistent name for
fault_process_userfaultfd().
__do_userfault()? :)
Quoted 4 lines
Signature · 3 lines
Cheers, David
Signature · 2 lines
Sincerely yours, Mike.
David Hildenbrand (Arm)
maintainerQuoted 10 lines
Quoted 8 lines
Quoted 5 lines
-
/* folio not in the pagecache, try to allocate */ if (IS_ERR(folio))
folio =__kvm_gmem_folio_alloc(inode, vmf->pgoff);That looks better in general. We should likely find a better/more consistent name for
fault_process_userfaultfd().
__do_userfault()? :)
Yes, that matches the naming scheme :)
Signature · 3 lines
Cheers, David