mm, kvm: allow uffd support in guest_memfd
Subsystems (10)
Mike Rapoport
maintainerv2 cover mm, kvm: allow uffd support in guest_memfd
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in guest_memfd.
As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.
In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.
Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
The handling of UFFDIO_MISSING in guest_memfd requires ability to remove a folio from page cache, the best way I could find was exporting filemap_remove_folio() to KVM.
I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.
Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.
v1 changes:
* instead of returning uffd-specific values from ->fault() handlers add __do_userfault() helper to resolve user faults in __do_fault()
* address comments from Peter
* rebased on v7.0-c1
RFC: https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (11):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_get_pmd() helper.
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic(): remove retry logic
Nikita Kalyazin (3):
KVM: guest_memfd: implement userfaultfd operations
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd
Peter Xu (1):
mm: generalize handling of userfaults in __do_fault()
include/linux/mm.h | 5 + include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 73 +- mm/filemap.c | 1 + mm/hugetlb.c | 15 + mm/memory.c | 43 ++ mm/shmem.c | 188 ++--- mm/userfaultfd.c | 692 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 84 ++- 10 files changed, 858 insertions(+), 448 deletions(-)
base-commit: 6de23f81a5e08be8fbf5e8d7e9febc72a5b5f27f --
2.51.0
Mike Rapoport
maintainerPATCH v2 1/15 userfaultfd: introduce mfill_copy_folio_locked() helper
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().
This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.
No functional change.
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/* +* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A +* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr); + +if (ret) +return -EFAULT ; + +flush_dcache_folio (folio); +return ret; +} + static int mfill_atomic_pte_copy(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,uffd_flags_t flags, struct folio **foliop) { - void *kaddr; int ret;struct folio *folio;
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,if (!folio) goto out; - kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (kaddr, (const void __user *) src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (kaddr); + ret =mfill_copy_folio_locked (folio, src_addr);/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* don't free the page */ goto out; } - -flush_dcache_folio (folio); }else { folio = *foliop; *foliop =NULL ;
Signature · 1 line
2.51.0
Re: [PATCH v2 01/15] userfaultfd: introduce mfill_copy_folio_locked() helper
Quoted 5 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from
mfill_atomic_pte_copy()into a helper functionmfill_copy_folio_locked().
Quoted 8 lines
This makes improves code readability and makes complex
mfill_atomic_pte_copy()function easier to comprehend.No functional change.
Acked-by Peter Xu <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>---
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerPATCH v2 2/15 userfaultfd: introduce struct mfill_state
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic() passes a lot of parameters down to its callees.
Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.
Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.
The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.
--- mm/userfaultfd.c | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-)
mm/userfaultfd.c +82 -66
@@ -20,6 +20,20 @@#include "internal.h" #include "swap.h" +struct mfill_state { +struct userfaultfd_ctx *ctx; +unsigned long src_start; +unsigned long dst_start; +unsigned long len; +uffd_flags_t flags; + +struct vm_area_struct *vma; +unsigned long src_addr; +unsigned long dst_addr; +struct folio *folio; +pmd_t *pmd; +}; +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_atomic_pte_copy (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +unsigned long src_addr = state->src_addr; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio; +int ret; -if (!*foliop) { +if (!state->folio) { ret = -ENOMEM ; folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) { ret = -ENOENT ; - *foliop = folio; + state->folio = folio;/* don't free the page */ goto out; } }else { - folio = *foliop; - *foliop =NULL ; + folio = state->folio; + state->folio =NULL ; }/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,return ret; } -static int mfill_atomic_pte_zeropage (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_zeropage (struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl;int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, }/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ -static int mfill_atomic_pte_continue (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_continue (struct mfill_state *state) { -struct inode *inode =file_inode (dst_vma->vm_file); +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct inode *inode =file_inode (dst_vma->vm_file); +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio;struct page *page;int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ -static int mfill_atomic_pte_poison (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_poison (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma;struct mm_struct *dst_mm = dst_vma->vm_mm; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl; +int ret; _dst_pte =make_pte_marker (PTE_MARKER_POISONED ); ret = -EAGAIN ;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, uffd_flags_t flags);#endif /* CONFIG_HUGETLB_PAGE */ -static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long src_addr = state->src_addr; +unsigned long dst_addr = state->dst_addr; +struct folio **foliop = &state->folio; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;ssize_t err; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { -return mfill_atomic_pte_continue (dst_pmd, dst_vma, - dst_addr, flags); - }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) { -return mfill_atomic_pte_poison (dst_pmd, dst_vma, - dst_addr, flags); - } +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) +return mfill_atomic_pte_continue (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) +return mfill_atomic_pte_poison (state);/* * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,*/ if (!(dst_vma->vm_flags &VM_SHARED )) {if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); + err =mfill_atomic_pte_copy (state);else - err =mfill_atomic_pte_zeropage (dst_pmd, - dst_vma, dst_addr); + err =mfill_atomic_pte_zeropage (state); }else { err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long len, uffd_flags_t flags) { +struct mfill_state state = (struct mfill_state ){ + .ctx = ctx, + .dst_start = dst_start, + .src_start = src_start, + .flags = flags, + + .src_addr = src_start, + .dst_addr = dst_start, + };struct mm_struct *dst_mm = ctx->mm;struct vm_area_struct *dst_vma; +long copied =0 ;ssize_t err;pmd_t *dst_pmd; -unsigned long src_addr, dst_addr; -long copied; -struct folio *folio;/* * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); - src_addr = src_start; - dst_addr = dst_start; - copied =0 ; - folio =NULL ; retry:/* * Make sure the vma is not shared, that the dst range is
@@ -790,12 +804,14 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; -while (src_addr < src_start + len) { -pmd_t dst_pmdval; + state.vma = dst_vma; -VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); + +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;
@@ -827,34 +843,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,* tables under us; pte_offset_map_lock() will deal with that. */ - err =mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, - src_addr, flags, &folio); + state.pmd = dst_pmd; + err =mfill_atomic_pte (&state);cond_resched ();if (unlikely (err == -ENOENT )) {void *kaddr;up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); -VM_WARN_ON_ONCE (!folio); +uffd_mfill_unlock (state.vma); +VM_WARN_ON_ONCE (!state.folio); - kaddr =kmap_local_folio (folio,0 ); + kaddr =kmap_local_folio (state.folio,0 ); err =copy_from_user (kaddr, - (const void __user *) src_addr, + (const void __user *)state.src_addr,PAGE_SIZE );kunmap_local (kaddr);if (unlikely (err)) { err = -EFAULT ; goto out; } -flush_dcache_folio (folio); +flush_dcache_folio (state.folio); goto retry; }else -VM_WARN_ON_ONCE (folio); +VM_WARN_ON_ONCE (state.folio);if (!err) { - dst_addr +=PAGE_SIZE ; - src_addr +=PAGE_SIZE ; + state.dst_addr +=PAGE_SIZE ; + state.src_addr +=PAGE_SIZE ; copied +=PAGE_SIZE ;if (fatal_signal_pending (current))
@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.51.0
Re: [PATCH v2 02/15] userfaultfd: introduce struct mfill_state
Quoted 17 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>---
[...]
Quoted 5 lines
mm/userfaultfd.c +0
-0if (fatal_signal_pending (current))@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock);
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 891
Quoted 10 lines
mm/userfaultfd.c +3
-3-uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
I guess, "copied" could be easily avoided by
comparing state.src_addr with start_addr.
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerRe: [PATCH v2 02/15] userfaultfd: introduce struct mfill_state
Quoted 24 lines · David Hildenbrand (Arm)
Quoted 17 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>---
[...]
Quoted 3 lines
mm/userfaultfd.c +0
-0if (fatal_signal_pending (current))@@ -866,10 +882,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
Quoted 16 lines
Quoted 12 lines
mm/userfaultfd.c +3
-3out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);Quoted 1 line
return copied ? copied : err;I guess, "copied" could be easily avoided by
comparing state.src_addr withstart_addr.
Yeah, but a dedicated variable is more robust against potential changes in state handling. And it's surely more descriptive and shorter than
state.src_addr - src_addr;
Quoted 1 line
Thanks!
Quoted 4 lines
Signature · 3 lines
Cheers, David
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 3/15 userfaultfd: introduce mfill_get_pmd() helper.
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.
Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().
--- mm/userfaultfd.c | 103 ++++++++++++++++++++++++----------------------- 1 file changed, 53 insertions(+), 50 deletions(-)
mm/userfaultfd.c +53 -50
@@ -157,6 +157,57 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) +{ +pgd_t *pgd; +p4d_t *p4d; +pud_t *pud; + + pgd =pgd_offset (mm, address); + p4d =p4d_alloc (mm, pgd, address); +if (!p4d) +return NULL ; + pud =pud_alloc (mm, p4d, address); +if (!pud) +return NULL ; +/* +* Note that we didn't run this because the pmd was +* missing, the *pmd may be already established and in +* turn it may also be a trans_huge_pmd. +*/ +return pmd_alloc (mm, pud, address); +} + +static int mfill_get_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd; +pmd_t dst_pmdval; + + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +if (unlikely (pmd_none (dst_pmdval)) && +unlikely (__pte_alloc (dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_trans_huge (dst_pmdval))) +return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +} +/* Check if dst_addr is outside of file's size. Must be called with ptl held. */ static bool mfill_file_over_size (struct vm_area_struct *dst_vma,unsigned long dst_addr)
@@ -489,27 +540,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)return ret; } -static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) -{ -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; - - pgd =pgd_offset (mm, address); - p4d =p4d_alloc (mm, pgd, address); -if (!p4d) -return NULL ; - pud =pud_alloc (mm, p4d, address); -if (!pud) -return NULL ; -/* -* Note that we didn't run this because the pmd was -* missing, the *pmd may be already established and in -* turn it may also be a trans_huge_pmd. -*/ -return pmd_alloc (mm, pud, address); -} -#ifdef CONFIG_HUGETLB_PAGE /* * mfill_atomic processing for HUGETLB vmas. Note that this routine is
@@ -742,7 +772,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err; -pmd_t *dst_pmd;/* * Sanitize the command parameters:
@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_get_pmd (&state); +if (err)break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -if (unlikely (pmd_none (dst_pmdval)) && -unlikely (__pte_alloc (dst_mm, dst_pmd))) { - err = -ENOMEM ; -break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -/* -* If the dst_pmd is THP don't override it and just be strict. -* (This includes the case where the PMD used to be THP and -* changed back to none after __pte_alloc().) -*/ -if (unlikely (!pmd_present (dst_pmdval) || -pmd_trans_huge (dst_pmdval))) { - err = -EEXIST ; -break ; - } -if (unlikely (pmd_bad (dst_pmdval))) { - err = -EFAULT ; -break ; - }/* * For shmem mappings, khugepaged is allowed to remove page * tables under us; pte_offset_map_lock() will deal with that. */ - state.pmd = dst_pmd; err =mfill_atomic_pte (&state);cond_resched ();
Signature · 1 line
2.51.0
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.
Quoted 1 line · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Nit: "." at the end of the patch subject
Quoted 38 lines
There is a lengthy code chunk in
mfill_atomic()that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of
mfill_atomic_pte_copy().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 103 ++++++++++++++++++++++++----------------------- 1 file changed, 53 insertions(+), 50 deletions(-)mm/userfaultfd.c +16
-0@@ -157,6 +157,57 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) +{ +pgd_t *pgd; +p4d_t *p4d; +pud_t *pud; + + pgd =pgd_offset (mm, address); + p4d =p4d_alloc (mm, pgd, address); +if (!p4d) +return NULL ; + pud =pud_alloc (mm, p4d, address); +if (!pud) +return NULL ; +/* +* Note that we didn't run this because the pmd was +* missing, the *pmd may be already established and in
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 185
Quoted 10 lines
mm/userfaultfd.c +10
-0+* turn it may also be a trans_huge_pmd. +*/ +return pmd_alloc (mm, pud, address); +} + +static int mfill_get_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd; +pmd_t dst_pmdval;
I'd just have both on a single line.
Quoted 7 lines
mm/userfaultfd.c +7
-0+ + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); + if (unlikely(pmd_none(dst_pmdval)) &&
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 202
Quoted 10 lines
mm/userfaultfd.c +10
-0+ unlikely(__pte_alloc(dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_trans_huge (dst_pmdval)))
Can we directly switch to pmd_leaf() while touching that?
Quoted 7 lines
mm/userfaultfd.c +7
-0+return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +}
[...]
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 842
Quoted 13 lines
mm/userfaultfd.c +2
-5/* * Sanitize the command parameters: @@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_get_pmd (&state); +if (err)
It's a bit odd that a "get" function doesn't return a PMD pointer but instead stores it in the state.
Maybe more like "mfill_prepare_pmd" ? But actually you want to have a pte table.
mfill_prepare_pte_table() or alternatively mfill_alloc_pte_table() / mfill_alloc_dst_pte_table()
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerRe: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.
Quoted 4 lines · David Hildenbrand (Arm)
Quoted 1 line · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Nit: "." at the end of the patch subject
Oops :/
Quoted 7 lines
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 185
Quoted 5 lines
mm/userfaultfd.c +5
-0+static int mfill_get_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd; +pmd_t dst_pmdval;I'd just have both on a single line.
Can do :)
Quoted 8 lines
mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 202
Quoted 6 lines
mm/userfaultfd.c +6
-0+/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_trans_huge (dst_pmdval)))Can we directly switch to
pmd_leaf()while touching that?
You mean instead of pmd_trans_huge()?
Yeah, sure.
Quoted 16 lines
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, · line 842
Quoted 11 lines
mm/userfaultfd.c +2
-5@@ -809,41 +838,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_get_pmd (&state); +if (err)It's a bit odd that a "get" function doesn't return a PMD pointer but instead stores it in the state.
Quoted 5 lines
Maybe more like "
mfill_prepare_pmd" ? But actually you want to have a pte table.
mfill_prepare_pte_table()or alternativelymfill_alloc_pte_table()/mfill_alloc_dst_pte_table()
As it actually allocates the pte table once in 512 times, I'd prefer mfill_establish_pmd().
Quoted 3 lines
Signature · 2 lines
Cheers, David
Signature · 2 lines
Sincerely yours, Mike.
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 03/15] userfaultfd: introduce mfill_get_pmd() helper.
Quoted 6 lines
Quoted 3 lines
mfill_prepare_pte_table()or alternativelymfill_alloc_pte_table()/mfill_alloc_dst_pte_table()As it actually allocates the pte table once in 512 times, I'd prefer
mfill_establish_pmd().
Replying only now (spotted v3 in my inbox)m sounds good to me.
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerPATCH v2 4/15 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.
This function will be used later during refactoring of mfill_atomic_pte_copy().
Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.
--- mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 51 deletions(-)
mm/userfaultfd.c +73 -51
@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) + goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) + goto out; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock; + +out: + state->vma = dst_vma; +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;
@@ -768,8 +835,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, .src_addr = src_start, .dst_addr = dst_start, }; -struct mm_struct *dst_mm = ctx->mm; -struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err;
@@ -784,57 +849,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (dst_start + len <= dst_start); retry: -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ - dst_vma =uffd_mfill_lock (dst_mm, dst_start, len); -if (IS_ERR (dst_vma)) { - err =PTR_ERR (dst_vma); + err =mfill_get_vma (&state); +if (err) goto out; - } - -/* -* If memory mappings are changing because of non-cooperative -* operation (e.g. mremap) running in parallel, bail out and -* request the user to retry later -*/ -down_read (&ctx->map_changing_lock); - err = -EAGAIN ; -if (atomic_read (&ctx->mmap_changing)) - goto out_unlock; - - err = -EINVAL ; -/* -* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but -* it will overwrite vm_ops, so vma_is_anonymous must return false. -*/ -if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && - dst_vma->vm_flags &VM_SHARED )) - goto out_unlock; - -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) - goto out_unlock;/* * If this is a HUGETLB vma, pass off to appropriate routine */ -if (is_vm_hugetlb_page (dst_vma)) -return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, +if (is_vm_hugetlb_page (state.vma)) +return mfill_atomic_hugetlb (ctx, state.vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; - - state.vma = dst_vma; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);
@@ -853,8 +878,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (unlikely (err == -ENOENT )) {void *kaddr; -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state);VM_WARN_ON_ONCE (!state.folio); kaddr =kmap_local_folio (state.folio,0 );
@@ -883,9 +907,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -out_unlock: -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);
Signature · 1 line
2.51.0
A message that is not in the archive (sent off-list or not yet received): abe1FHyYinvfLYnw@hyeyoo
Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
[Adding missing mailing lists to Cc that I omitted by mistake]
Quoted 131 lines · Harry Yoo
Quoted 6 lines · Mike Rapoport (Microsoft)
Split the code that finds, locks and verifies VMA from
mfill_atomic()into a helper function.This function will be used later during refactoring of
mfill_atomic_pte_copy().Quoted 6 lines
Add a counterpart
mfill_put_vma()helper that unlocks the VMA and releasesmap_changing_lock.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Peter Xu <[email protected]>---
Apparently this patch seems to have two issues... Fortunately, it did not land mainline yet and can be addressed.
Deepanshu and Edward sent fixes, but it should be fixed as part of v3 (if Mike plans to do so) or as a fix-up on patch 4.
Please keep in mind that my understanding of this patchset is limited. I'm just doing some mechanical analysis.
Quoted 22 lines
mm/userfaultfd.c | 124 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 51 deletions(-)mm/userfaultfd.c +11
-0@@ -157,6 +157,73 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags;mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 180
Quoted 10 lines
mm/userfaultfd.c +10
-0+struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma);
state->lenis always initialized to zero in patch 2. syzbot triggered a warning infolio_add_new_anon_rmap(), which appears to be due to failing to verify the range inuffd_mfill_lock()(at least syzbot says it's fixed [1]).[1] https://lore.kernel.org/linux-mm/[email protected]
It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.
At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).
Quoted 25 lines
mm/userfaultfd.c +25
-0+/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP ))mm/userfaultfd.c static void uffd_mfill_unlock(struct vm_area_struct *vma) · line 216
Quoted 10 lines
mm/userfaultfd.c +10
-0+ goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) + goto out; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock;In case of error,
mfill_put_vma()is supposed to unlock appropriate locks butstate->vmais not set, so it does nothing.Pointed out by Edward Adam Davis [3]
[3] https://lore.kernel.org/linux-mm/[email protected]I think, this should probably be fine in patch 4, but patch 5 adds
if (!state->vma) return;, leading to unreleased locks on error paths.Quoted 13 lines
mm/userfaultfd.c +10
-0+ +out: + state->vma = dst_vma; +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Quoted 11 lines · Harry Yoo
Quoted 11 lines
It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.
At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).
Harry,
You are correct that once vm_refcnt > 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ in vma_start_read(), and CHECK 2 can miss this if mmap_write_unlock() completes before CHECK 2 runs.
Here is the exact race:
vma_start_read():
/* CHECK 1 */
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
goto err;
/*
* RACE WINDOW: vm_refcnt is still 0 here!
* UFFDIO_UNREGISTER can run:
*
* mmap_write_lock() -> mm_lock_seq = 11
* vma_start_write(vma) -> vm_lock_seq = 11
* __split_vma() -> vma->vm_end = 0x4ca000
* mmap_write_unlock() -> mm_lock_seq = 12
*
* writer completes entirely before vm_refcnt++!
*/
__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...);
/* vm_refcnt = 1 now, but vma->vm_end already modified! */
/* CHECK 2 */
if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq)))
/*
* vm_lock_seq(11) == mm_lock_seq(12)?
* NO! writer already finished and unlocked!
* mm_lock_seq incremented to 12 (even=unlocked)
* CHECK 2 MISSES the race!
*/
return vma;
/*
* returns split vma with vm_end=0x4ca000
* but vm_refcnt=1 (lock held)
*/Now mfill_atomic loop runs with split vma:
while (state.src_addr < src_start + len) {
/* iteration 1 to N: dst_addr = 0x1b1000 to 0x4c9000
* all within vma->vm_end(0x4ca000)
*/
/* iteration N+1: dst_addr = 0x4ca000 */
err = mfill_atomic_pte(&state);
mfill_atomic_install_pte(state->vma, dst_addr=0x4ca000)
folio_add_new_anon_rmap(vma, 0x4ca000)
VM_WARN_ON_ONCE(address < vma->vm_start ||
address + (nr << 12) > vma->vm_end);
/* 0x4ca000 >= vma->vm_end(0x4ca000) -> WARN! */
}Without my fix:
CRASH at folio_add_new_anon_rmap
With my fix:
if (state.dst_addr < state.vma->vm_start ||
state.dst_addr >= state.vma->vm_end) {
mfill_put_vma(&state);
state.dst_start = state.dst_addr;
state.len = dst_start + len - state.dst_addr;
err = mfill_get_vma(&state);
if (err)
break;
}
/* catches split, re-lookups correct VMA safely */So both fixes are needed:
Harry's fix (state.len):
fixes state.len uninitialized
mfill_get_vma validates correct range
in the first call before loop
My fix (bounds check):
catches split VMA that slipped
through CHECK 2 during loop
because writer finished before
CHECK 2 ran
Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Quoted 34 lines · Deepanshu Kartikey
Quoted 11 lines · Harry Yoo
Quoted 11 lines
It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.
At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).
Harry,
You are correct that once
vm_refcnt> 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ invma_start_read(), and CHECK 2 can miss this ifmmap_write_unlock()completes before CHECK 2 runs.Here is the exact race:
vma_start_read():/* CHECK 1 */ if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) goto err; /* * RACE WINDOW: vm_refcnt is still 0 here! * UFFDIO_UNREGISTER can run: * * mmap_write_lock() -> mm_lock_seq = 11 * vma_start_write(vma) -> vm_lock_seq = 11
Quoted 8 lines
* __split_vma() -> vma->vm_end = 0x4ca000
* mmap_write_unlock() -> mm_lock_seq = 12
*
* writer completes entirely before vm_refcnt++!
*/
__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...);
/* vm_refcnt = 1 now, but vma->vm_end already modified! */It is true that vma->vm_end might have changed before acquiring the vma lock, but it doesn't matter as long as you verify the range after acquiring the lock, no? (that's what uffd_mfill_lock() does)
You're not really supposed to read vma->vm_end before acquiring the vma lock and use the value because nothing guarantees that the VMA is stable until the lock is acquired.
Or am I still missing something?
Quoted 13 lines
/* CHECK 2 */
if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq)))
/*
* vm_lock_seq(11) == mm_lock_seq(12)?
* NO! writer already finished and unlocked!
* mm_lock_seq incremented to 12 (even=unlocked)
* CHECK 2 MISSES the race!
*/
return vma;
/*
* returns split vma with vm_end=0x4ca000
* but vm_refcnt=1 (lock held)
*/Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH v2 4/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Quoted 49 lines · Harry Yoo
Quoted 34 lines · Deepanshu Kartikey
Quoted 11 lines · Harry Yoo
Quoted 11 lines
It seems there's another attempt to fix the syzbot report from Deepanshu Kartikey [2], which I didn't take a deeper look.
At first look [2] looks a bit wrong way to fix to me though, because it allows operating only on a single VMA nothing should really split or shrink the VMA if somebody is holding the VMA lock in read mode (and the validation of the range is done while holding the lock).
Harry,
You are correct that once
vm_refcnt> 0, nobody can split the VMA. However the split can happen in the race window BEFORE vm_refcnt++ invma_start_read(), and CHECK 2 can miss this ifmmap_write_unlock()completes before CHECK 2 runs.Here is the exact race:
vma_start_read():/* CHECK 1 */ if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) goto err; /* * RACE WINDOW: vm_refcnt is still 0 here! * UFFDIO_UNREGISTER can run: * * mmap_write_lock() -> mm_lock_seq = 11 * vma_start_write(vma) -> vm_lock_seq = 11Quoted 8 lines
* __split_vma() -> vma->vm_end = 0x4ca000 * mmap_write_unlock() -> mm_lock_seq = 12 * * writer completes entirely before vm_refcnt++! */ __refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, ...); /* vm_refcnt = 1 now, but vma->vm_end already modified! */It is true that
vma->vm_endmight have changed before acquiring the vma lock, but it doesn't matter as long as you verify the range after acquiring the lock, no? (that's whatuffd_mfill_lock()does)
Quoted 6 lines
You're not really supposed to read
vma->vm_endbefore acquiring the vma lock and use the value because nothing guarantees that the VMA is stable until the lock is acquired.Or am I still missing something?
Harry, you are right. The real bug is state.len = 0. I withdraw my fix.
Thank you for the explanation
Mike Rapoport
maintainerPATCH v2 5/15 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).
In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.
Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.
As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.
--- mm/userfaultfd.c | 78 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 24 deletions(-)
mm/userfaultfd.c +54 -24
@@ -159,6 +159,9 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma)static void mfill_put_vma (struct mfill_state *state) { +if (!state->vma) +return ; +up_read (&state->ctx->map_changing_lock);uffd_mfill_unlock (state->vma); state->vma =NULL ;
@@ -404,35 +407,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_get_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) { +/* +* Fallback to copy_from_user outside mmap_lock. +* If retry is successful, mfill_copy_folio_locked() returns +* with locks retaken by mfill_get_vma(). +* If there was an error, we must mfill_put_vma() anyway and it +* will take care of unlocking if needed. +*/ + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto out_release; }/*
@@ -442,17 +473,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio); - ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;folio_put (folio); goto out; }
Signature · 1 line
2.51.0
Mike Rapoport
maintainerPATCH v2 6/15 userfaultfd: move vma_can_userfault out of line
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault() has grown pretty big and it's not called on performance critical path.
Move it out of line.
No functional changes.
--- include/linux/userfaultfd_k.h | 35 ++--------------------------------- mm/userfaultfd.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags, -bool wp_async) -{ - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true; - -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && - !vma_is_anonymous (vma)) -return false; - -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async);static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {
mm/userfaultfd.c +33 -0
@@ -2020,6 +2020,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,return moved ? moved : err; } +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async) +{ + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + +if ((vm_flags &VM_UFFD_MINOR ) && + (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) +return false; + +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem & hugetlbfs are not supported but only +* anonymous. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && + !vma_is_anonymous (vma)) +return false; + +/* By default, allow any of anon|shmem|hugetlb */ +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +static void userfaultfd_set_vm_flags (struct vm_area_struct *vma,vm_flags_t vm_flags) {
Signature · 1 line
2.51.0
Mike Rapoport
maintainerPATCH v2 7/15 userfaultfd: introduce vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.
Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.
Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.
Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.
--- include/linux/mm.h | 5 +++++ include/linux/userfaultfd_k.h | 6 ++++++ mm/hugetlb.c | 15 +++++++++++++++ mm/shmem.c | 15 +++++++++++++++ mm/userfaultfd.c | 36 ++++++++++++++++++++++++++--------- 5 files changed, 68 insertions(+), 9 deletions(-)
include/linux/mm.h +5 -0
@@ -741,6 +741,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -826,6 +828,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;
mm/hugetlb.c +15 -0
@@ -4818,6 +4818,18 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops hugetlb_uffd_ops = { + .can_userfault = hugetlb_can_userfault, +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4831,6 +4843,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +15 -0
@@ -3294,6 +3294,15 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,shmem_inode_unacct_blocks (inode,1 ); return ret; } + +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops shmem_uffd_ops = { + .can_userfault = shmem_can_userfault, +};#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS
@@ -5313,6 +5322,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5322,6 +5334,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)
mm/userfaultfd.c +27 -9
@@ -34,6 +34,25 @@ struct mfill_state {pmd_t *pmd; }; +static bool anon_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +/* anonymous memory does not support MINOR mode */ +if (vm_flags &VM_UFFD_MINOR ) +return false; +return true; +} + +static const struct vm_uffd_ops anon_uffd_ops = { + .can_userfault = anon_can_userfault, +}; + +static const struct vm_uffd_ops *vma_uffd_ops (struct vm_area_struct *vma) +{ +if (vma_is_anonymous (vma)) +return &anon_uffd_ops; +return vma->vm_ops ? vma->vm_ops->uffd_ops :NULL ; +} +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {
@@ -2023,13 +2042,15 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags,bool wp_async) { - vm_flags &= __VM_UFFD_FLAGS; +const struct vm_uffd_ops *ops =vma_uffd_ops (vma); -if (vma->vm_flags &VM_DROPPABLE ) +/* only VMAs that implement vm_uffd_ops are supported */ +if (!ops)return false; -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE )return false;/*
@@ -2041,16 +2062,13 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,/* * If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. +* uffd-wp, then only anonymous memory is supported */ if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma))return false; -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); +return ops->can_userfault (vma, vm_flags); } static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
Signature · 1 line
2.51.0
Mike Rapoport
maintainerRe: [PATCH v2 07/15] userfaultfd: introduce vm_uffd_ops
Review
Quoted 10 lines · Mike Rapoport
bool vma_can_userfault(struct
vm_area_struct*vma,vm_flags_tvm_flags,(quoted diff) +3
-2bool wp_async) { - vm_flags &= __VM_UFFD_FLAGS; + const struct vm_uffd_ops *ops = vma_uffd_ops(vma); - if (vma->vm_flags & VM_DROPPABLE) + /* only VMAs that implement vm_uffd_ops are supported */ + if (!ops) return false;
Just found out that rejecting a VMA that does not have uffd_ops but was registered in WP-only mode with WP_ASYNC uffd context breaks pagemap_ioctl() test and more broadly it breaks tracking of writes in SysV shared memory areas.
This is weird that it's possible to use uffd with SysV SHM, but it's out there for some time and I afraid we can't change that :/
Andrew, can you apply this as a fixup please
From 6e3319ceab93d84558e735e1f4f451e80c85b267 Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (Microsoft)" <[email protected]>
Date: Wed, 11 Mar 2026 20:21:38 +0200
Subject: [PATCH 1/1] userfaultfd: allow registration of WP_ASYNC for any VMA
Registration of a VMA with WP_ASYNC userfaulfd context in write-protect mode does not require any VMA-specific resolution of user faults and these faults are completely handled by the generic page fault handler.
This functionality existed since the introduction of WP_ASYNC mode and it allows tracking writes to SysV shared memory mappings (shmget(2) and shmat(2)).
Move the check for WP mode before checking for presence of ->uffd_ops in a VMA to restore the original behaviour.
--- mm/userfaultfd.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
mm/userfaultfd.c +8 -8
@@ -2044,22 +2044,22 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, {const struct vm_uffd_ops *ops =vma_uffd_ops (vma); -/* only VMAs that implement vm_uffd_ops are supported */ -if (!ops) -return false; - vm_flags &= __VM_UFFD_FLAGS; -if (vma->vm_flags &VM_DROPPABLE ) -return false; -/* -* If wp async enabled, and WP is the only mode enabled, allow any +* If WP is the only mode enabled and context is wp async, allow any * memory type. */ if (wp_async && (vm_flags ==VM_UFFD_WP ))return true; +/* For any other mode reject VMAs that don't implement vm_uffd_ops */ +if (!ops) +return false; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; +/* * If user requested uffd-wp but not enabled pte markers for * uffd-wp, then only anonymous memory is supported
Signature · 1 line
2.51.0
Mike Rapoport
maintainerPATCH v2 8/15 shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.
Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.
--- include/linux/userfaultfd_k.h | 7 +++++++ mm/shmem.c | 15 ++++++++++++++- mm/userfaultfd.c | 32 ++++++++++++++++---------------- 3 files changed, 37 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* +* Called to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +* The returned folio is locked and with reference held. +*/ +struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); };/* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, return ret; } +static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +} +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) {return true; }static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .can_userfault = shmem_can_userfault, + .get_folio_noalloc = shmem_get_folio_noalloc, };#endif /* CONFIG_USERFAULTFD */
mm/userfaultfd.c +16 -16
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)struct userfaultfd_ctx *ctx = state->ctx;uffd_flags_t flags = state->flags;struct vm_area_struct *dst_vma; +const struct vm_uffd_ops *ops;int err;/*
@@ -231,10 +232,12 @@ static int mfill_get_vma(struct mfill_state *state)if (is_vm_hugetlb_page (dst_vma)) goto out; -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + ops =vma_uffd_ops (dst_vma); +if (!ops) goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && + !ops->get_folio_noalloc) goto out_unlock; out:
@@ -577,6 +580,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)static int mfill_atomic_pte_continue (struct mfill_state *state) {struct vm_area_struct *dst_vma = state->vma; +const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);
@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (!ops) +return -EOPNOTSUPP ; + + folio = ops->get_folio_noalloc (inode, pgoff);/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; - } +if (IS_ERR_OR_NULL (folio)) +return -EFAULT ; page =folio_file_page (folio, pgoff);if (PageHWPoison (page)) {
@@ -609,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.51.0
Re: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
Quoted 11 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with
UFFDIO_CONTINUEit needs to get a folio that already exists in the pagecache backing that VMA.Instead of using
shmem_get_folio()for that, add aget_folio_noalloc()method to 'structvm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Quoted 5 lines
Implement
get_folio_noalloc()method for shmem and slightly refactor userfaultfd'smfill_get_vma()andmfill_atomic_pte_continue()to support this new API.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
I found myself asking why HugeTLB wasn't included here; it's because we call into hugetlb_mfill_atomic_pte() pretty early. It would be nice to have HugeTLB implement get_folio_noalloc() and use the same path as shmem, but we don't have to do that now.
Feel free to add:
Quoted 93 lines
--- include/linux/userfaultfd_k.h | 7 +++++++ mm/shmem.c | 15 ++++++++++++++- mm/userfaultfd.c | 32 ++++++++++++++++---------------- 3 files changed, 37 insertions(+), 17 deletions(-)include/linux/userfaultfd_k.h +7
-0@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* +* Called to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +* The returned folio is locked and with reference held. +*/ +struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); };/* A combined operation mode + behavior flags. */ mm/shmem.c +14
-1@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, return ret; } +static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +} +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) {return true; }static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .can_userfault = shmem_can_userfault, + .get_folio_noalloc = shmem_get_folio_noalloc, };#endif /* CONFIG_USERFAULTFD */ mm/userfaultfd.c +7
-3@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)struct userfaultfd_ctx *ctx = state->ctx;uffd_flags_t flags = state->flags;struct vm_area_struct *dst_vma; +const struct vm_uffd_ops *ops;int err;/* @@ -231,10 +232,12 @@ static int mfill_get_vma(struct mfill_state *state)if (is_vm_hugetlb_page (dst_vma)) goto out; -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + ops =vma_uffd_ops (dst_vma); +if (!ops) goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && + !ops->get_folio_noalloc) goto out_unlock; out:@@ -577,6 +580,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)static int mfill_atomic_pte_continue (struct mfill_state *state) {struct vm_area_struct *dst_vma = state->vma; +const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);
mm/userfaultfd.c static int mfill_atomic_pte_continue(struct mfill_state *state) · line 594
Quoted 6 lines
mm/userfaultfd.c +2
-1@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (!ops) +return -EOPNOTSUPP ;
Perhaps WARN_ON_ONCE(!ops)?
Quoted 36 lines
mm/userfaultfd.c +7
-13+ + folio = ops->get_folio_noalloc (inode, pgoff);/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; - } +if (IS_ERR_OR_NULL (folio)) +return -EFAULT ; page =folio_file_page (folio, pgoff);if (PageHWPoison (page)) {@@ -609,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ --2.51.0
Andrew Morton
maintainerRe: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
Quoted 18 lines · On Thu, 26 Mar 2026 16:43:27 -0700 James Houghton <[email protected]> wrote:
Quoted 11 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with
UFFDIO_CONTINUEit needs to get a folio that already exists in the pagecache backing that VMA.Instead of using
shmem_get_folio()for that, add aget_folio_noalloc()method to 'structvm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.Quoted 5 lines
Implement
get_folio_noalloc()method for shmem and slightly refactor userfaultfd'smfill_get_vma()andmfill_atomic_pte_continue()to support this new API.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
Quoted 8 lines
I found myself asking why HugeTLB wasn't included here; it's because we call into
hugetlb_mfill_atomic_pte()pretty early. It would be nice to have HugeTLB implementget_folio_noalloc()and use the same path as shmem, but we don't have to do that now.Feel free to add:
Reviewed-by James Houghton <[email protected]>
Thanks.
Quoted 11 lines
Quoted 11 lines
static int
mfill_atomic_pte_continue(struct mfill_state *state){ struct vm_area_struct *dst_vma = state->vma;mm/userfaultfd.c +1
-0+const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);@@ -586,16 +590,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret;
Quoted 6 lines
mm/userfaultfd.c static int mfill_atomic_pte_continue(struct mfill_state *state) · line 594
Quoted 3 lines
mm/userfaultfd.c +2
-1- ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (!ops) +return -EOPNOTSUPP ;Perhaps
WARN_ON_ONCE(!ops)?
fyi, I (and I expect many others) tend to consider a "Reviewed-by:" as end-of-message, So if you have additional comments then add something like "comments below" so we know to keep scrolling!
If there are no additional comments, simply zap all quoted material after the R-b to remove all doubt.
Mike Rapoport
maintainerRe: [PATCH v2 08/15] shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
In reply to James HoughtonHi James,
Quoted 16 lines · James Houghton
Quoted 15 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with
UFFDIO_CONTINUEit needs to get a folio that already exists in the pagecache backing that VMA.Instead of using
shmem_get_folio()for that, add aget_folio_noalloc()method to 'structvm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.Implement
get_folio_noalloc()method for shmem and slightly refactor userfaultfd'smfill_get_vma()andmfill_atomic_pte_continue()to support this new API.
Quoted 6 lines
Quoted 1 line
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>I found myself asking why HugeTLB wasn't included here; it's because we call into
hugetlb_mfill_atomic_pte()pretty early. It would be nice to have HugeTLB implementget_folio_noalloc()and use the same path as shmem, but we don't have to do that now.
You are welcome to join the effort ;-)
Quoted 3 lines
Feel free to add:
Reviewed-by James Houghton <[email protected]>
Thanks!
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 9/15 userfaultfd: introduce vm_uffd_ops->alloc_folio()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio
Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.
Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.
Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should return allocate a and return folio or NULL if allocation fails. +*/ +struct folio *(*alloc_folio)(struct vm_area_struct *vma, +unsigned long addr); };/* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44
142 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v2 09/15] userfaultfd: introduce vm_uffd_ops->alloc_folio()
Quoted 15 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor
mfill_atomic_pte_zeroed_folio()andmfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio()andmfill_atomic_pte_copy()perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folioSplit a
__mfill_atomic_pte()helper that handles both cases and uses newly introducedvm_uffd_ops->alloc_folio()to allocate the folio.
Quoted 7 lines
Pass the ops structure from the callers to
__mfill_atomic_pte()to later allow usinganon_uffd_opsforMAP_PRIVATEmappings of file-backed VMAs.Note, that the new ops method is called
alloc_folio()rather thanfolio_alloc()to avoid clash withalloc_tagmacrofolio_alloc().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
Feel free to add:
Quoted 10 lines
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)include/linux/userfaultfd_k.h +0
-0
include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 96
Quoted 6 lines
include/linux/userfaultfd_k.h +3
-0@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should return allocate a and return folio or NULL if allocation fails.
"Should allocate and return a folio or NULL if allocation fails."
I see this mistake is fixed in the next patch. :)
Quoted 73 lines
include/linux/userfaultfd_k.h +3
-0+*/ +struct folio *(*alloc_folio)(struct vm_area_struct *vma, +unsigned long addr); };/* A combined operation mode + behavior flags. */ mm/userfaultfd.c +23
-10@@ -42,8 +42,26 @@ static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)return true; } +static struct folio *anon_alloc_folio (struct vm_area_struct *vma, +unsigned long addr) +{ +struct folio *folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, + addr); + +if (!folio) +return NULL ; + +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) { +folio_put (folio); +return NULL ; + } + +return folio; +} +static const struct vm_uffd_ops anon_uffd_ops = { .can_userfault = anon_can_userfault, + .alloc_folio = anon_alloc_folio, };static const struct vm_uffd_ops *vma_uffd_ops (struct vm_area_struct *vma)@@ -458,7 +476,8 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioreturn 0 ; } -static int mfill_atomic_pte_copy (struct mfill_state *state) +static int __mfill_atomic_pte (struct mfill_state *state, +const struct vm_uffd_ops *ops) {unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;@@ -466,16 +485,12 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)struct folio *folio;int ret; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); + folio = ops->alloc_folio (state->vma, state->dst_addr);if (!folio)return -ENOMEM ; - ret = -ENOMEM ; -if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_copy_folio_locked (folio, src_addr); -if (unlikely (ret)) { +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { + ret =mfill_copy_folio_locked (folio, src_addr);/* * Fallback to copy_from_user outside mmap_lock. * If retry is successful, mfill_copy_folio_locked() returns @@ -483,9 +498,15 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)* If there was an error, we must mfill_put_vma() anyway and it * will take care of unlocking if needed. */ - ret =mfill_copy_folio_retry (state, folio); -if (ret)
mm/userfaultfd.c static int mfill_atomic_pte_copy(struct mfill_state *state) · line 509
Quoted 10 lines
mm/userfaultfd.c +9
-1- goto out_release; +if (unlikely (ret)) { + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto err_folio_put; + } + }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) { +clear_user_highpage (&folio->page, state->dst_addr); + }else { +VM_WARN_ONCE (1 ,"unknown UFFDIO operation" );
"Unknown UFFDIO operation. flags=%x", flags
seems a little better to me.
Quoted 78 lines
mm/userfaultfd.c +16
-34}/* @@ -498,47 +519,30 @@ static int mfill_atomic_pte_copy(struct mfill_state *state) ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) - goto out_release; -out: -return ret; -out_release: + goto err_folio_put; + +return 0 ; + +err_folio_put: +folio_put (folio);/* Don't return -ENOENT so that our caller won't retry */ if (ret == -ENOENT ) ret = -EFAULT ; -folio_put (folio); - goto out; +return ret; } -static int mfill_atomic_pte_zeroed_folio (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct folio *folio; -int ret = -ENOMEM ; - - folio =vma_alloc_zeroed_movable_folio (dst_vma, dst_addr); -if (!folio) -return ret; - -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_put; +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -/* -* The memory barrier inside __folio_mark_uptodate makes sure that -* zeroing out the folio become visible before mapping the page -* using set_pte_at(). See do_anonymous_page(). -*/ -__folio_mark_uptodate (folio); +return __mfill_atomic_pte (state, ops); +} - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true,0 ); -if (ret) - goto out_put; +static int mfill_atomic_pte_zeroed_folio (struct mfill_state *state) +{ +const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); -return 0 ; -out_put: -folio_put (folio); -return ret; +return __mfill_atomic_pte (state, ops); }static int mfill_atomic_pte_zeropage (struct mfill_state *state)@@ -551,7 +555,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)int ret;if (mm_forbids_zeropage (dst_vma->vm_mm)) -return mfill_atomic_pte_zeroed_folio (dst_pmd, dst_vma, dst_addr); +return mfill_atomic_pte_zeroed_folio (state); _dst_pte =pte_mkspecial (pfn_pte (my_zero_pfn (dst_addr), dst_vma->vm_page_prot)); --2.51.0
Mike Rapoport
maintainerRe: [PATCH v2 09/15] userfaultfd: introduce vm_uffd_ops->alloc_folio()
Quoted 19 lines · James Houghton
Quoted 18 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor
mfill_atomic_pte_zeroed_folio()andmfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio()andmfill_atomic_pte_copy()perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folioSplit a
__mfill_atomic_pte()helper that handles both cases and uses newly introducedvm_uffd_ops->alloc_folio()to allocate the folio.Pass the ops structure from the callers to
__mfill_atomic_pte()to later allow usinganon_uffd_opsforMAP_PRIVATEmappings of file-backed VMAs.
Quoted 8 lines
Quoted 4 lines
Note, that the new ops method is called
alloc_folio()rather thanfolio_alloc()to avoid clash withalloc_tagmacrofolio_alloc().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Feel free to add:
Reviewed-by James Houghton <[email protected]>
Thanks!
Quoted 19 lines
Quoted 10 lines
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)include/linux/userfaultfd_k.h +0
-0include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 96
Quoted 6 lines
include/linux/userfaultfd_k.h +3
-0@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should return allocate a and return folio or NULL if allocation fails. "Should allocate and return a folio or NULL if allocation fails."
Quoted 1 line
I see this mistake is fixed in the next patch. :)
Endless rebases :)
Will try to sort it out this time :)
Quoted 19 lines
Quoted 6 lines
include/linux/userfaultfd_k.h +0
-2@@ -483,9 +498,15 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)* If there was an error, we must mfill_put_vma() anyway and it * will take care of unlocking if needed. */ - ret =mfill_copy_folio_retry (state, folio); -if (ret)include/linux/userfaultfd_k.h static int mfill_atomic_pte_copy(struct mfill_state *state) · line 509
Quoted 10 lines
include/linux/userfaultfd_k.h +9
-1- goto out_release; +if (unlikely (ret)) { + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto err_folio_put; + } + }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) { +clear_user_highpage (&folio->page, state->dst_addr); + }else { +VM_WARN_ONCE (1 ,"unknown UFFDIO operation" );"Unknown UFFDIO operation. flags=%x", flags
Quoted 1 line
seems a little better to me.
Yeah, why not.
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 10/15 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.
Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().
Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c
mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 21 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 156 deletions(-)
include/linux/shmem_fs.h +0 -14
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +15 -6
@@ -93,10 +93,24 @@ struct vm_uffd_ops {struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff);/* * Called during resolution of UFFDIO_COPY request. -* Should return allocate a and return folio or NULL if allocation fails. +* Should allocate and return a folio or NULL if allocation +* fails. */ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache. +* Returns 0 on success, error code on failure. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);
mm/shmem.c +53 -95
177 lines not on this page · raw message
mm/userfaultfd.c +38 -41
148 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v2 10/15] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 56 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Add
filemap_add()andfilemap_remove()methods tovm_uffd_opsand use them in__mfill_atomic_pte()to add shmem folios to page cache and remove them in case of error.Implement these methods in shmem along with
vm_uffd_ops->alloc_folio()and dropshmem_mfill_atomic_pte().Since userfaultfd now does not reference any functions from shmem, drop include if
linux/shmem_fs.hfrommm/userfaultfd.c
mfill_atomic_install_pte()is not used anywhere outside of mm/userfaultfd, make it static.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 21 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 156 deletions(-)include/linux/shmem_fs.h +0
-14@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that include/linux/userfaultfd_k.h +0
-0@@ -93,10 +93,24 @@ struct vm_uffd_ops {struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff);/*
include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 103
Quoted 10 lines
include/linux/userfaultfd_k.h +5
-1* Called during resolution of UFFDIO_COPY request. -* Should return allocate a and return folio or NULL if allocation fails. +* Should allocate and return a folio or NULL if allocation +* fails. */ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache.
I don't think "should lock the folio" is accurate. That sounds like "it will call folio_lock()" but it actually calls __folio_set_locked(). Maybe this is better:
"Should only be called with a folio returned by alloc_folio() above. The folio will set to locked."
Quoted 246 lines
include/linux/userfaultfd_k.h +10
-5+* Returns 0 on success, error code on failure. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */ @@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);mm/shmem.c +53
-95@@ -3181,118 +3181,73 @@ static struct inode *shmem_get_inode(struct mnt_idmap *idmap,#endif /* CONFIG_TMPFS_QUOTA */ #ifdef CONFIG_USERFAULTFD -int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) -{ -struct inode *inode =file_inode (dst_vma->vm_file); -struct shmem_inode_info *info =SHMEM_I (inode); +static struct folio *shmem_mfill_folio_alloc (struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file);struct address_space *mapping = inode->i_mapping; +struct shmem_inode_info *info =SHMEM_I (inode); +pgoff_t pgoff =linear_page_index (vma, addr);gfp_t gfp =mapping_gfp_mask (mapping); -pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); -void *page_kaddr;struct folio *folio; -int ret; -pgoff_t max_off; - -if (shmem_inode_acct_blocks (inode,1 )) { -/* -* We may have got a page, returned -ENOENT triggering a retry, -* and now we find ourselves with -ENOMEM. Release the page, to -* avoid a BUG_ON in our caller. -*/ -if (unlikely (*foliop)) { -folio_put (*foliop); - *foliop =NULL ; - } -return -ENOMEM ; - } -if (!*foliop) { - ret = -ENOMEM ; - folio =shmem_alloc_folio (gfp,0 , info, pgoff); -if (!folio) - goto out_unacct_blocks; +if (unlikely (pgoff >=DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ))) +return NULL ; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { - page_kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (page_kaddr, - (const void __user *)src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (page_kaddr); - -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - *foliop = folio; - ret = -ENOENT ; -/* don't free the page */ - goto out_unacct_blocks; - } + folio =shmem_alloc_folio (gfp,0 , info, pgoff); +if (!folio) +return NULL ; -flush_dcache_folio (folio); - }else {/* ZEROPAGE */ -clear_user_highpage (&folio->page, dst_addr); - } - }else { - folio = *foliop; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - *foliop =NULL ; +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) { +folio_put (folio); +return NULL ; } -VM_BUG_ON (folio_test_locked (folio)); -VM_BUG_ON (folio_test_swapbacked (folio)); +return folio; +} + +static int shmem_mfill_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +gfp_t gfp =mapping_gfp_mask (mapping); +int err; +__folio_set_locked (folio);__folio_set_swapbacked (folio); -__folio_mark_uptodate (folio); - - ret = -EFAULT ; - max_off =DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ); -if (unlikely (pgoff >= max_off)) - goto out_release; - ret =mem_cgroup_charge (folio, dst_vma->vm_mm, gfp); -if (ret) - goto out_release; - ret =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); -if (ret) - goto out_release; + err =shmem_add_to_page_cache (folio, mapping, pgoff,NULL , gfp); +if (err) + goto err_unlock; - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - &folio->page, true, flags); -if (ret) - goto out_delete_from_cache; +if (shmem_inode_acct_blocks (inode,1 )) { + err = -ENOMEM ; + goto err_delete_from_cache; + } +folio_add_lru (folio);shmem_recalc_inode (inode,1 ,0 ); -folio_unlock (folio); +return 0 ; -out_delete_from_cache: + +err_delete_from_cache:filemap_remove_folio (folio); -out_release: +err_unlock: +folio_unlock (folio); +return err; +} + +static void shmem_mfill_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (vma->vm_file); + +filemap_remove_folio (folio); +shmem_recalc_inode (inode,0 ,0 );folio_unlock (folio); -folio_put (folio); -out_unacct_blocks: -shmem_inode_unacct_blocks (inode,1 ); -return ret; }static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff)@@ -3315,6 +3270,9 @@ static bool shmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags)static const struct vm_uffd_ops shmem_uffd_ops = { .can_userfault = shmem_can_userfault, .get_folio_noalloc = shmem_get_folio_noalloc, + .alloc_folio = shmem_mfill_folio_alloc, + .filemap_add = shmem_mfill_filemap_add, + .filemap_remove = shmem_mfill_filemap_remove, };#endif /* CONFIG_USERFAULTFD */ mm/userfaultfd.c +4
-8@@ -14,7 +14,6 @@#include <linux/userfaultfd_k.h> #include <linux/mmu_notifier.h> #include <linux/hugetlb.h> -#include <linux/shmem_fs.h> #include <asm/tlbflush.h> #include <asm/tlb.h> #include "internal.h" @@ -340,10 +339,10 @@ static bool mfill_file_over_size(struct vm_area_struct *dst_vma,* This function handles both MCOPY_ATOMIC_NORMAL and _CONTINUE for both shmem * and anon, and for both shared and private VMAs. */ -int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags) +static int mfill_atomic_install_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr,struct page *page, +uffd_flags_t flags) {int ret;struct mm_struct *dst_mm = dst_vma->vm_mm;@@ -387,9 +386,6 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, goto out_unlock;if (page_in_cache) { -/* Usually, cache pages are already added to LRU */ -if (newly_allocated) -folio_add_lru (folio);folio_add_file_rmap_pte (folio, page, dst_vma); }else {folio_add_new_anon_rmap (folio, dst_vma, dst_addr,RMAP_EXCLUSIVE );
mm/userfaultfd.c int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 404
Quoted 5 lines
mm/userfaultfd.c +2
-0@@ -404,6 +400,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte); +if (page_in_cache) +folio_unlock (folio);
I don't really like doing the folio_unlock() *here*, I think it's clearer if the callers (mfill_atomic_pte_continue() and __mfill_atomic_pte()) unlocked the folio themselves. But that's just my opinion.
Quoted 81 lines
mm/userfaultfd.c +27
-10+/* No need to invalidate - it was non-present before */ update_mmu_cache (dst_vma, dst_addr, dst_pte); ret =0 ;@@ -516,13 +515,22 @@ static int __mfill_atomic_pte(struct mfill_state *state,*/ __folio_mark_uptodate (folio); +if (ops->filemap_add) { + ret = ops->filemap_add (folio, state->vma, state->dst_addr); +if (ret) + goto err_folio_put; + } + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, - &folio->page, true, flags); + &folio->page, flags);if (ret) - goto err_folio_put; + goto err_filemap_remove;return 0 ; +err_filemap_remove: +if (ops->filemap_remove) + ops->filemap_remove (folio, state->vma); err_folio_put:folio_put (folio);/* Don't return -ENOENT so that our caller won't retry */ @@ -535,6 +543,18 @@ static int mfill_atomic_pte_copy(struct mfill_state *state) {const struct vm_uffd_ops *ops =vma_uffd_ops (state->vma); +/* +* The normal page fault path for a MAP_PRIVATE mapping in a +* file-backed VMA will invoke the fault, fill the hole in the file and +* COW it right away. The result generates plain anonymous memory. +* So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll +* generate anonymous memory directly without actually filling the +* hole. For the MAP_PRIVATE case the robustness check only happens in +* the pagetable (to verify it's still none) and not in the page cache. +*/ +if (!(state->vma->vm_flags &VM_SHARED )) + ops = &anon_uffd_ops; +return __mfill_atomic_pte (state, ops); }@@ -554,7 +574,8 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)spinlock_t *ptl;int ret; -if (mm_forbids_zeropage (dst_vma->vm_mm)) +if (mm_forbids_zeropage (dst_vma->vm_mm) || + (dst_vma->vm_flags &VM_SHARED ))return mfill_atomic_pte_zeroed_folio (state); _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr),@@ -609,11 +630,10 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) } ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, - page, false, flags); + page, flags);if (ret) goto out_release; -folio_unlock (folio);return 0 ; out_release:@@ -836,41 +856,18 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma; -unsigned long src_addr = state->src_addr; -unsigned long dst_addr = state->dst_addr; -struct folio **foliop = &state->folio;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;
mm/userfaultfd.c extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 868
Quoted 10 lines
mm/userfaultfd.c +4
-1-ssize_t err;if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ))return mfill_atomic_pte_continue (state);if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON ))return mfill_atomic_pte_poison (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) +return mfill_atomic_pte_copy (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) +return mfill_atomic_pte_zeropage (state);
Thanks for this cleanup. :)
Quoted 14 lines
mm/userfaultfd.c +0
-13-/* -* The normal page fault path for a shmem will invoke the -* fault, fill the hole in the file and COW it right away. The -* result generates plain anonymous memory. So when we are -* asked to fill an hole in a MAP_PRIVATE shmem mapping, we'll -* generate anonymous memory directly without actually filling -* the hole. For the MAP_PRIVATE case the robustness check -* only happens in the pagetable (to verify it's still none) -* and not in the radix tree. -*/ -if (!(dst_vma->vm_flags &VM_SHARED )) { -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (state);
mm/userfaultfd.c extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 870
Quoted 10 lines
mm/userfaultfd.c +1
-9-else - err =mfill_atomic_pte_zeropage (state); - }else { - err =shmem_mfill_atomic_pte (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); - } - -return err; +return -EOPNOTSUPP ;
WARN_ONCE() here I think.
Feel free to add:
Mike Rapoport
maintainerRe: [PATCH v2 10/15] shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
Quoted 72 lines · James Houghton
Quoted 56 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Add
filemap_add()andfilemap_remove()methods tovm_uffd_opsand use them in__mfill_atomic_pte()to add shmem folios to page cache and remove them in case of error.Implement these methods in shmem along with
vm_uffd_ops->alloc_folio()and dropshmem_mfill_atomic_pte().Since userfaultfd now does not reference any functions from shmem, drop include if
linux/shmem_fs.hfrommm/userfaultfd.c
mfill_atomic_install_pte()is not used anywhere outside of mm/userfaultfd, make it static.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 21 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 79 +++++++++--------- 4 files changed, 106 insertions(+), 156 deletions(-)include/linux/shmem_fs.h +0
-14@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that include/linux/userfaultfd_k.h +0
-0@@ -93,10 +93,24 @@ struct vm_uffd_ops {struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff);/* include/linux/userfaultfd_k.h struct vm_uffd_ops { · line 103
Quoted 10 lines
include/linux/userfaultfd_k.h +5
-1* Called during resolution of UFFDIO_COPY request. -* Should return allocate a and return folio or NULL if allocation fails. +* Should allocate and return a folio or NULL if allocation +* fails. */ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should lock the folio and add it to VMA's page cache. I don't think "should lock the folio" is accurate. That sounds like "it will call
folio_lock()" but it actually calls__folio_set_locked(). Maybe this is better:
Quoted 2 lines
"Should only be called with a folio returned by
alloc_folio()above. The folio will set to locked."
Yeah, sounds good.
Quoted 9 lines
Quoted 9 lines
include/linux/userfaultfd_k.h +4
-0+* Returns 0 on success, error code on failure. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr);@@ -404,6 +400,9 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte);
Quoted 7 lines
include/linux/userfaultfd_k.h int mfill_atomic_install_pte(pmd_t *dst_pmd, · line 404
Quoted 2 lines
include/linux/userfaultfd_k.h +2
-0+if (page_in_cache) +folio_unlock (folio);I don't really like doing the
folio_unlock()*here*, I think it's clearer if the callers (mfill_atomic_pte_continue()and__mfill_atomic_pte()) unlocked the folio themselves. But that's just my opinion.
We already have page_in_cache here, so I'd prefer to keep a single folio_unlock() rather than add additional if (page_in_cache) in __mfill_atomic_pte().
Quoted 52 lines
Quoted 14 lines
include/linux/userfaultfd_k.h +1
-5+/* No need to invalidate - it was non-present before */ update_mmu_cache (dst_vma, dst_addr, dst_pte); ret =0 ;@@ -836,41 +856,18 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma; -unsigned long src_addr = state->src_addr; -unsigned long dst_addr = state->dst_addr; -struct folio **foliop = &state->folio;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;include/linux/userfaultfd_k.h extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 868
Quoted 10 lines
include/linux/userfaultfd_k.h +4
-1-ssize_t err;if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ))return mfill_atomic_pte_continue (state);if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON ))return mfill_atomic_pte_poison (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) +return mfill_atomic_pte_copy (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) +return mfill_atomic_pte_zeropage (state);Thanks for this cleanup. :)
Quoted 14 lines
include/linux/userfaultfd_k.h +0
-13-/* -* The normal page fault path for a shmem will invoke the -* fault, fill the hole in the file and COW it right away. The -* result generates plain anonymous memory. So when we are -* asked to fill an hole in a MAP_PRIVATE shmem mapping, we'll -* generate anonymous memory directly without actually filling -* the hole. For the MAP_PRIVATE case the robustness check -* only happens in the pagetable (to verify it's still none) -* and not in the radix tree. -*/ -if (!(dst_vma->vm_flags &VM_SHARED )) { -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (state);include/linux/userfaultfd_k.h extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, · line 869
Quoted 10 lines
include/linux/userfaultfd_k.h +1
-9-else - err =mfill_atomic_pte_zeropage (state); - }else { - err =shmem_mfill_atomic_pte (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); - } - -return err; +return -EOPNOTSUPP ;
Quoted 1 line
WARN_ONCE()here I think.
I'll add VM_WARN_ONCE() here.
Quoted 3 lines
Feel free to add:
Reviewed-by James Houghton <[email protected]>
Thanks!
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 11/15 userfaultfd: mfill_atomic(): remove retry logic
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().
Drop the retry logic from mfill_atomic().
--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
@@ -29,7 +29,6 @@ struct mfill_state {struct vm_area_struct *vma;unsigned long src_addr;unsigned long dst_addr; -struct folio *folio;pmd_t *pmd; };
@@ -898,7 +897,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); -retry: err =mfill_get_vma (&state);if (err) goto out;
@@ -925,26 +923,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =mfill_atomic_pte (&state);cond_resched (); -if (unlikely (err == -ENOENT )) { -void *kaddr; - -mfill_put_vma (&state); -VM_WARN_ON_ONCE (!state.folio); - - kaddr =kmap_local_folio (state.folio,0 ); - err =copy_from_user (kaddr, - (const void __user *)state.src_addr, -PAGE_SIZE ); -kunmap_local (kaddr); -if (unlikely (err)) { - err = -EFAULT ; - goto out; - } -flush_dcache_folio (state.folio); - goto retry; - }else -VM_WARN_ON_ONCE (state.folio); -if (!err) { state.dst_addr +=PAGE_SIZE ; state.src_addr +=PAGE_SIZE ;
@@ -959,8 +937,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,mfill_put_vma (&state); out: -if (state.folio) -folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.51.0
Mike Rapoport
maintainerPATCH v2 12/15 mm: generalize handling of userfaults in __do_fault()
In reply to Mike RapoportFrom: Peter Xu <[email protected]>
When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:
- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist
Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().
For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.
Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.
--- mm/memory.c | 43 +++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 12 ------------ mm/userfaultfd.c | 8 ++++++++ 3 files changed, 51 insertions(+), 12 deletions(-)
mm/memory.c +43 -0
@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +#ifdef CONFIG_USERFAULTFD +static vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode; +struct folio *folio; + +if (!(userfaultfd_missing (vma) ||userfaultfd_minor (vma))) +return 0 ; + + inode =file_inode (vma->vm_file); + folio = vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff); +if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ +folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) +return handle_userfault (vmf,VM_UFFD_MINOR ); + }else { +if (userfaultfd_missing (vma)) +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +#else +static inline vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value.
@@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/* +* If this is an userfaultfd trap, process it in advance before +* triggering the genuine fault handler. +*/ + ret =__do_userfault (vmf); +if (ret) +return ret; + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY |VM_FAULT_DONE_COW )))
mm/shmem.c +0 -12
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {
mm/userfaultfd.c +8 -0
@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !vma_is_anonymous(vma))return false; +/* +* File backed memory with PTE level mappigns must implement +* ops->get_folio_noalloc() +*/ +if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && + !ops->get_folio_noalloc) +return false; +return ops->can_userfault (vma, vm_flags); }
Signature · 1 line
2.51.0
Re: [PATCH v2 12/15] mm: generalize handling of userfaults in __do_fault()
Quoted 41 lines · Mike Rapoport
From: Peter Xu <[email protected]>
When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call
handle_userfault()with appropriate mode:-
VM_UFFD_MINORif VMA is registered in minor mode and the folio exists
-VM_UFFD_MISSINGif VMA is registered in missing mode and the folio does not existInstead of calling
handle_userfault()directly from a specific ->fault() handler, call__do_userfault()helper from the generic__do_fault().For VMAs registered with userfaultfd the new
__do_userfault()helper will check if the folio is found in the page cache usingvm_uffd_ops->get_folio_noalloc()and callhandle_userfault()with the appropriate mode.Make
vm_uffd_ops->get_folio_noalloc()required method for non-anonymous VMAs mapped at PTE level.Signed-off-by Peter Xu <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/memory.c | 43 +++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 12 ------------ mm/userfaultfd.c | 8 ++++++++ 3 files changed, 51 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +#ifdef CONFIG_USERFAULTFD +static vm_fault_t __do_userfault (struct vm_fault *vmf)
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5342
Quoted 10 lines
mm/memory.c +10
-0+{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode; +struct folio *folio; + +if (!(userfaultfd_missing (vma) ||userfaultfd_minor (vma))) +return 0 ; + + inode =file_inode (vma->vm_file); + folio = vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff);
If you do away with the change you made to vma_can_userfault(), please add a WARN_ON_ONCE() here if uffd_ops or uffd_ops->get_folio_noalloc are not present.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5347
Quoted 5 lines
mm/memory.c +5
-0+if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/
I think a whole new op is better than adding a flag. Something like uffd_ops->folio_present().
Quoted 22 lines
mm/memory.c +18
-0+folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) + return handle_userfault(vmf, VM_UFFD_MINOR); + }else { +if (userfaultfd_missing (vma)) +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +#else +static inline vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value.
mm/memory.c static vm_fault_t __do_fault(struct vm_fault *vmf) · line 5399
Quoted 5 lines
mm/memory.c +2
-0@@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/* +* If this is an userfaultfd trap, process it in advance before
"If this is a userfault"
Quoted 44 lines
mm/memory.c +6
-0+* triggering the genuine fault handler. +*/ + ret =__do_userfault (vmf); +if (ret) +return ret; + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY |VM_FAULT_DONE_COW )))mm/shmem.c +0
-12@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {mm/userfaultfd.c +0
-0
mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2049
Quoted 5 lines
mm/userfaultfd.c +2
-0@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !vma_is_anonymous(vma))return false; +/* +* File backed memory with PTE level mappigns must implement
"File-backed VMAs (except HugeTLB VMAs) must implement ops->get_folio_alloc() to support userfaults, as all userfaults in file-backed VMAs wlll call ops->get_folio_alloc() to determine the userfault type."
mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2053
Quoted 4 lines
mm/userfaultfd.c +4
-0+* ops->get_folio_noalloc() +*/ +if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && + !ops->get_folio_noalloc)
With the separate folio_present() (or whatever) op, this check becomes more obvious. (Looking at this right now, it looks like we're checking for UFFDIO_CONTINUE/minor fault support here, but that's not really what's going on.)
Honestly I'm not 100% sure if this check should be here. If it's going to be here, then we should probably also have checks like these here:
if (vm_flags & VM_UFFD_MINOR && !ops->get_folio_noalloc)
and
if (vm_flags & VM_UFFD_MISSING && !ops->alloc_folio)
So, maybe we should just leave it up to ops->can_userfault() to make sure we're doing the right thing?
Quoted 8 lines
mm/userfaultfd.c +2
-1+return false; +return ops->can_userfault (vma, vm_flags); } --2.51.0
Mike Rapoport
maintainerRe: [PATCH v2 12/15] mm: generalize handling of userfaults in __do_fault()
Quoted 57 lines · James Houghton
Quoted 41 lines · Mike Rapoport
From: Peter Xu <[email protected]>
When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call
handle_userfault()with appropriate mode:-
VM_UFFD_MINORif VMA is registered in minor mode and the folio exists
-VM_UFFD_MISSINGif VMA is registered in missing mode and the folio does not existInstead of calling
handle_userfault()directly from a specific ->fault() handler, call__do_userfault()helper from the generic__do_fault().For VMAs registered with userfaultfd the new
__do_userfault()helper will check if the folio is found in the page cache usingvm_uffd_ops->get_folio_noalloc()and callhandle_userfault()with the appropriate mode.Make
vm_uffd_ops->get_folio_noalloc()required method for non-anonymous VMAs mapped at PTE level.Signed-off-by Peter Xu <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/memory.c | 43 +++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 12 ------------ mm/userfaultfd.c | 8 ++++++++ 3 files changed, 51 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -5328,6 +5328,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +#ifdef CONFIG_USERFAULTFD +static vm_fault_t __do_userfault (struct vm_fault *vmf)mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5342
Quoted 10 lines
mm/memory.c +10
-0+{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode; +struct folio *folio; + +if (!(userfaultfd_missing (vma) ||userfaultfd_minor (vma))) +return 0 ; + + inode =file_inode (vma->vm_file); + folio = vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff);If you do away with the change you made to
vma_can_userfault(), please add aWARN_ON_ONCE()here ifuffd_opsoruffd_ops->get_folio_noallocare not present.
Quoted 8 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5347
Quoted 5 lines
mm/memory.c +5
-0+if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ I think a whole new op is better than adding a flag. Something like
uffd_ops->folio_present().
Right now it follows what shmem does, let's keep it as TODO for future optimizations.
Quoted 25 lines
Quoted 25 lines
mm/memory.c +18
-0+folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) + return handle_userfault(vmf, VM_UFFD_MINOR); + }else { +if (userfaultfd_missing (vma)) +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +#else +static inline vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value. @@ -5360,6 +5395,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; }
Quoted 4 lines
mm/memory.c static vm_fault_t __do_fault(struct vm_fault *vmf) · line 5399
Quoted 2 lines
mm/memory.c +2
-0+/* +* If this is an userfaultfd trap, process it in advance before "If this is a userfault"
Indeed :)
Quoted 47 lines
Quoted 47 lines
mm/memory.c +6
-0+* triggering the genuine fault handler. +*/ + ret =__do_userfault (vmf); +if (ret) +return ret; + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY |VM_FAULT_DONE_COW )))mm/shmem.c +0
-12@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {mm/userfaultfd.c +0
-0@@ -2045,6 +2045,14 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !vma_is_anonymous(vma))return false;
Quoted 7 lines
mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2049
Quoted 2 lines
mm/userfaultfd.c +2
-0+/* +* File backed memory with PTE level mappigns must implement "File-backed VMAs (except HugeTLB VMAs) must implement
ops->get_folio_alloc()to support userfaults, as all userfaults in file-backed VMAs wlll callops->get_folio_alloc()to determine the userfault type."
I like (except HugeTLB) more than "PTE ...", but it's not here to determine the userfaulfd type, it's here to ensure we can __do_userfault() for a memory type.
/*
* File backed VMAs (except HugeTLB) must implement
* ops->get_folio_noalloc() because it's required by
* __do_userfault() in page fault handling. */
Quoted 13 lines
mm/userfaultfd.c bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, · line 2053
Quoted 4 lines
mm/userfaultfd.c +4
-0+* ops->get_folio_noalloc() +*/ +if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && + !ops->get_folio_noalloc)With the separate
folio_present()(or whatever) op, this check becomes more obvious. (Looking at this right now, it looks like we're checking for UFFDIO_CONTINUE/minor fault support here, but that's not really what's going on.)Honestly I'm not 100% sure if this check should be here. If it's going to be here, then we should probably also have checks like these here:
Quoted 8 lines
if (
vm_flags&VM_UFFD_MINOR&& !ops->get_folio_noalloc)and
if (
vm_flags&VM_UFFD_MISSING&& !ops->alloc_folio)So, maybe we should just leave it up to
ops->can_userfault()to make sure we're doing the right thing?
This must be in the core so it can reject VMAs that can't be used in __do_userfault() without complicating hot path in the page fault handler with extra conditions.
Quoted 9 lines
Quoted 9 lines
mm/userfaultfd.c +2
-1+return false; +return ops->can_userfault (vma, vm_flags); } --2.51.0
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 13/15 KVM: guest_memfd: implement userfaultfd operations
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").
To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.
Add implementation of vm_uffd_ops to guest_memfd.
--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +82 -2
114 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v2 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 11 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit
7677f7fd8be7("userfaultfd: add minor fault registration mode").
Quoted 8 lines
To use the same mechanisms for VMs that use
guest_memfdto map their memory,guest_memfdshould support userfaultfd operations.Add implementation of
vm_uffd_opstoguest_memfd.Signed-off-by Nikita Kalyazin <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
Overall looks fine to me, but I am slightly concerned about in-place conversion[1], and I think you're going to want to implement a kvm_gmem_folio_present() op or something (like I was saying on the previous patch[2]).
[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVUJ5FL97d9ytxp2WXos6HS+U+ycpsi5VxffsW9vacr9Q@mail.gmail.com/
Some in-line comments below.
Quoted 30 lines
--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)mm/filemap.c +1
-0@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache virt/kvm/guest_memfd.c +1
-0@@ -7,6 +7,7 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h"
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 115
Quoted 8 lines
virt/kvm/guest_memfd.c +5
-0@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,return __kvm_gmem_prepare_folio (kvm, slot, index, folio); } +static struct folio *kvm_gmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +return __filemap_get_folio (inode->i_mapping, pgoff, +FGP_LOCK |FGP_ACCESSED ,0 ); +}
When in-place conversion is supported, I wonder what the semantics should be for when we get userfaults.
Upon a userspace access to a file offset that is populated but private, should we get a userfault or a SIGBUS?
I guess getting a userfault is strictly more useful for userspace, but I'm not sure which choice is more correct.
Quoted 55 lines
virt/kvm/guest_memfd.c +39
-2+/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest. @@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)* Fast-path: See if folio is already present in mapping to avoid * policy_lookup. */ - folio =__filemap_get_folio (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED ,0 ); + folio =kvm_gmem_get_folio_noalloc (inode, index);if (!IS_ERR (folio))return folio;@@ -457,12 +463,86 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, }#endif /* CONFIG_NUMA */ +#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +struct inode *inode =file_inode (vma->vm_file); + +/* +* Only support userfaultfd for guest_memfd with INIT_SHARED flag. +* This ensures the memory can be mapped to userspace. +*/ +if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )) +return false; + +return true; +} + +static struct folio *kvm_gmem_folio_alloc (struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); +struct mempolicy *mpol; +struct folio *folio; +gfp_t gfp; + +if (unlikely (pgoff >= (i_size_read (inode) >>PAGE_SHIFT ))) +return NULL ; + + gfp =mapping_gfp_mask (inode->i_mapping); + mpol =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, pgoff); + mpol = mpol ?:get_task_policy (current); + folio =filemap_alloc_folio (gfp,0 , mpol); +mpol_cond_put (mpol); + +return folio; +} + +static int kvm_gmem_filemap_add(struct folio *folio,
virt/kvm/guest_memfd.c static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, · line 512
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ struct vm_area_struct *vma, + unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +int err; + +__folio_set_locked (folio); + err =filemap_add_folio (mapping, folio, pgoff,GFP_KERNEL );
This is going to get more interesting with in-place conversion. I'm not really sure how to synchronize with it, but we'll probably need to take the invalidate lock for reading. And then we'll need a separate uffd_op to drop it after we install the PTE... I think.
Quoted 39 lines
virt/kvm/guest_memfd.c +27
-1+if (err) { +folio_unlock (folio); +return err; + } + +return 0 ; +} + +static void kvm_gmem_filemap_remove (struct folio *folio, +struct vm_area_struct *vma) +{ +filemap_remove_folio (folio); +folio_unlock (folio); +} + +static const struct vm_uffd_ops kvm_gmem_uffd_ops = { + .can_userfault = kvm_gmem_can_userfault, + .get_folio_noalloc = kvm_gmem_get_folio_noalloc, + .alloc_folio = kvm_gmem_folio_alloc, + .filemap_add = kvm_gmem_filemap_add, + .filemap_remove = kvm_gmem_filemap_remove, +}; +#endif /* CONFIG_USERFAULTFD */ +static const struct vm_operations_struct kvm_gmem_vm_ops = { .fault = kvm_gmem_fault_user_mapping, #ifdef CONFIG_NUMA .get_policy = kvm_gmem_get_policy, .set_policy = kvm_gmem_set_policy, #endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &kvm_gmem_uffd_ops, +#endif };static int kvm_gmem_mmap (struct file *file,struct vm_area_struct *vma) --2.51.0
Mike Rapoport
maintainerRe: [PATCH v2 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 17 lines · James Houghton
Quoted 16 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit
7677f7fd8be7("userfaultfd: add minor fault registration mode").To use the same mechanisms for VMs that use
guest_memfdto map their memory,guest_memfdshould support userfaultfd operations.Add implementation of
vm_uffd_opstoguest_memfd.
Quoted 8 lines
Quoted 3 lines
Signed-off-by Nikita Kalyazin <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Overall looks fine to me, but I am slightly concerned about in-place conversion[1], and I think you're going to want to implement a
kvm_gmem_folio_present()op or something (like I was saying on the previous patch[2]).
Let's solve each problem in it's time :)
Quoted 27 lines
[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVUJ5FL97d9ytxp2WXos6HS+U+ycpsi5VxffsW9vacr9Q@mail.gmail.com/Some in-line comments below.
Quoted 10 lines
--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)mm/filemap.c +0
-0mm/filemap.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 115
Quoted 8 lines
mm/filemap.c +5
-0@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,return __kvm_gmem_prepare_folio (kvm, slot, index, folio); } +static struct folio *kvm_gmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +return __filemap_get_folio (inode->i_mapping, pgoff, +FGP_LOCK |FGP_ACCESSED ,0 ); +}When in-place conversion is supported, I wonder what the semantics should be for when we get userfaults.
Quoted 5 lines
Upon a userspace access to a file offset that is populated but private, should we get a userfault or a SIGBUS?
I guess getting a userfault is strictly more useful for userspace, but I'm not sure which choice is more correct.
Me neither :)
We can deliver userfault, but just block UFFDIO_COPY, can't we?
Quoted 8 lines
Quoted 8 lines
mm/filemap.c +8
-0+static int kvm_gmem_filemap_add (struct folio *folio, +struct vm_area_struct *vma, +unsigned long addr) +{ +struct inode *inode =file_inode (vma->vm_file); +struct address_space *mapping = inode->i_mapping; +pgoff_t pgoff =linear_page_index (vma, addr); +int err;
Quoted 8 lines
mm/filemap.c
Quoted 3 lines
mm/filemap.c +3
-0+ +__folio_set_locked (folio); + err =filemap_add_folio (mapping, folio, pgoff,GFP_KERNEL );This is going to get more interesting with in-place conversion. I'm not really sure how to synchronize with it, but we'll probably need to take the invalidate lock for reading. And then we'll need a separate
uffd_opto drop it after we install the PTE... I think.
I think we can start simple and then move on along with the in-place conversion work. If there will be a need for a new uffd_ops callback we can add it then.
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 14/15 KVM: selftests: test userfaultfd minor for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
@@ -10,13 +10,17 @@#include <errno.h> #include <stdio.h> #include <fcntl.h> +#include <pthread.h> #include <linux/bitmap.h> #include <linux/falloc.h> #include <linux/sizes.h> +#include <linux/userfaultfd.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> #include "kvm_util.h" #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)close (fd1); } +struct fault_args { +char *addr; +char value; +}; + +static void *fault_thread_fn (void *arg) +{ +struct fault_args *args = arg; + +/* Trigger page fault */ + args->value = *args->addr; +return NULL ; +} + +static void test_uffd_minor (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_continue uffd_cont; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *mem_nofault, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xcd ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + +/* Map the guest_memfd twice: once with UFFD registered, once without */ + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + mem_nofault =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem_nofault !=MAP_FAILED ,"mmap should succeed" ); + +/* Register UFFD_MINOR on the first mapping */ + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MINOR ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + +/* +* Populate the page in the page cache first via mem_nofault. +* This is required for UFFD_MINOR - the page must exist in the cache. +* Write test data to the page. +*/ +memcpy (mem_nofault + offset, buf + offset, page_size); + +/* +* Now access the same page via mem (which has UFFD_MINOR registered). +* Since the page exists in the cache, this should trigger UFFD_MINOR. +*/ + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_MINOR , +"pagefault should be minor fault" ); + +/* Resolve the minor fault with UFFDIO_CONTINUE */ + uffd_cont.range.start = (unsigned long )fault_addr; + uffd_cont.range.len = page_size; + uffd_cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &uffd_cont); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_CONTINUE) should succeed" ); + +/* Wait for the faulting thread to complete */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* Verify the thread read the correct value */ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was written" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem_nofault, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (file_size, vm, flags);gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); + +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +gmem_test (uffd_minor, vm, flags); }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.51.0
Mike Rapoport
maintainerPATCH v2 15/15 KVM: selftests: test userfaultfd missing for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)close (uffd); } +static void test_uffd_missing (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_copy uffd_copy; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xab ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MISSING ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (!(msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ), +"pagefault should not be write-protect" ); + + uffd_copy.dst = (unsigned long )fault_addr; + uffd_copy.src = (unsigned long )(buf + offset); + uffd_copy.len = page_size; + uffd_copy.mode =0 ; + ret =ioctl (uffd,UFFDIO_COPY , &uffd_copy); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_COPY) should succeed" ); + +/* Wait for the faulting thread to complete - this provides the memory barrier */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* +* Now it's safe to check args.value - the thread has completed +* and memory is synchronized +*/ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was copied" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); -if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) {gmem_test (uffd_minor, vm, flags); +gmem_test (uffd_missing, vm, flags); + } }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.51.0
Andrew Morton
maintainerQuoted 1 line · On Fri, 6 Mar 2026 19:18:00 +0200 Mike Rapoport <[email protected]> wrote:
These patches enable support for userfaultfd in
guest_memfd.
Thanks, Mike. I'll get this into the pipeline for testing.
Review is a little patchy at this time but hopefully that will improve over the next few weeks.
Andrew Morton
maintainerQuoted 1 line · On Fri, 6 Mar 2026 19:18:00 +0200 Mike Rapoport <[email protected]> wrote:
These patches enable support for userfaultfd in
guest_memfd.
Review of this series is very thin at this time so can I please ask potential reviewers to spend some time with it?
To create additional time for review I have relocated this series to the tail of the mm-unstable queue. That probably means that movement into the mm-stable branch and upstreaming will occur in the second week of the upcoming (7.1-rc1) merge window.
Thanks.