mm, kvm: allow uffd support in guest_memfd
Subsystems (10)
Mike Rapoport
maintainerv4 cover mm, kvm: allow uffd support in guest_memfd
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches enable support for userfaultfd in guest_memfd.
As the ground work I refactored userfaultfd handling of PTE-based memory types (anonymous and shmem) and converted them to use vm_uffd_ops for allocating a folio or getting an existing folio from the page cache. shmem also implements callbacks that add a folio to the page cache after the data passed in UFFDIO_COPY was copied and remove the folio from the page cache if page table update fails.
In order for guest_memfd to notify userspace about page faults, there are new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler can return to inform the page fault handler that it needs to call handle_userfault() to complete the fault.
Nikita helped to plumb these new goodies into guest_memfd and provided basic tests to verify that guest_memfd works with userfaultfd.
The handling of UFFDIO_MISSING in guest_memfd requires ability to remove a folio from page cache, the best way I could find was exporting filemap_remove_folio() to KVM.
I deliberately left hugetlb out, at least for the most part. hugetlb handles acquisition of VMA and more importantly establishing of parent page table entry differently than PTE-based memory types. This is a different abstraction level than what vm_uffd_ops provides and people objected to exposing such low level APIs as a part of VMA operations.
Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed and I prefer to delay it until the dust settles after the changes in this set.
v4 changes:
* fix lock leak not only in patch 4, but also in patch 2, thanks Harry
v3: https://lore.kernel.org/all/[email protected]
* add fixes from Harry and Andrei
* fix handling of WP-only mode for WP_ASYNC contexts in vma_can_userfault()
* address David's comments about mfill_get_pmd() and rename it to mfill_establish_pmd()
* add VM_WARN()s for unsupported operations (James)
* update comments using James' suggestions
v2: https://lore.kernel.org/all/[email protected]
* instead of returning uffd-specific values from ->fault() handlers add __do_userfault() helper to resolve user faults in __do_fault()
* address comments from Peter
* rebased on v7.0-c1
RFC: https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (11):
userfaultfd: introduce mfill_copy_folio_locked() helper
userfaultfd: introduce struct mfill_state
userfaultfd: introduce mfill_establish_pmd() helper
userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
userfaultfd: retry copying with locks dropped in
mfill_atomic_pte_copy()
userfaultfd: move vma_can_userfault out of line
userfaultfd: introduce vm_uffd_ops
shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
userfaultfd: introduce vm_uffd_ops->alloc_folio()
shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
userfaultfd: mfill_atomic(): remove retry logic
Nikita Kalyazin (3):
KVM: guest_memfd: implement userfaultfd operations
KVM: selftests: test userfaultfd minor for guest_memfd
KVM: selftests: test userfaultfd missing for guest_memfd
Peter Xu (1):
mm: generalize handling of userfaults in __do_fault()
include/linux/mm.h | 5 + include/linux/shmem_fs.h | 14 - include/linux/userfaultfd_k.h | 73 +- mm/filemap.c | 1 + mm/hugetlb.c | 15 + mm/memory.c | 43 ++ mm/shmem.c | 188 ++--- mm/userfaultfd.c | 694 ++++++++++-------- .../testing/selftests/kvm/guest_memfd_test.c | 191 +++++ virt/kvm/guest_memfd.c | 84 ++- 10 files changed, 860 insertions(+), 448 deletions(-)
base-commit: c369299895a591d96745d6492d4888259b004a9e --
2.53.0
Mike Rapoport
maintainerPATCH v4 1/15 userfaultfd: introduce mfill_copy_folio_locked() helper
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split copying of data when locks held from mfill_atomic_pte_copy() into a helper function mfill_copy_folio_locked().
This makes improves code readability and makes complex mfill_atomic_pte_copy() function easier to comprehend.
No functional change.
--- mm/userfaultfd.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 24 deletions(-)
mm/userfaultfd.c +35 -24
@@ -238,6 +238,40 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd,return ret; } +static int mfill_copy_folio_locked (struct folio *folio,unsigned long src_addr) +{ +void *kaddr; +int ret; + + kaddr =kmap_local_folio (folio,0 ); +/* +* The read mmap_lock is held here. Despite the +* mmap_lock being read recursive a deadlock is still +* possible if a writer has taken a lock. For example: +* +* process A thread 1 takes read lock on own mmap_lock +* process A thread 2 calls mmap, blocks taking write lock +* process B thread 1 takes page fault, read lock on own mmap lock +* process B thread 2 calls mmap, blocks taking write lock +* process A thread 1 blocks taking read lock on process B +* process B thread 1 blocks taking read lock on process A +* +* Disable page faults to prevent potential deadlock +* and retry the copy outside the mmap_lock. +*/ +pagefault_disable (); + ret =copy_from_user (kaddr, (const void __user *) src_addr, +PAGE_SIZE ); +pagefault_enable (); +kunmap_local (kaddr); + +if (ret) +return -EFAULT ; + +flush_dcache_folio (folio); +return ret; +} + static int mfill_atomic_pte_copy(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,uffd_flags_t flags, struct folio **foliop) { - void *kaddr; int ret;struct folio *folio;
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,if (!folio) goto out; - kaddr =kmap_local_folio (folio,0 ); -/* -* The read mmap_lock is held here. Despite the -* mmap_lock being read recursive a deadlock is still -* possible if a writer has taken a lock. For example: -* -* process A thread 1 takes read lock on own mmap_lock -* process A thread 2 calls mmap, blocks taking write lock -* process B thread 1 takes page fault, read lock on own mmap lock -* process B thread 2 calls mmap, blocks taking write lock -* process A thread 1 blocks taking read lock on process B -* process B thread 1 blocks taking read lock on process A -* -* Disable page faults to prevent potential deadlock -* and retry the copy outside the mmap_lock. -*/ -pagefault_disable (); - ret =copy_from_user (kaddr, (const void __user *) src_addr, -PAGE_SIZE ); -pagefault_enable (); -kunmap_local (kaddr); + ret =mfill_copy_folio_locked (folio, src_addr);/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* don't free the page */ goto out; } - -flush_dcache_folio (folio); }else { folio = *foliop; *foliop =NULL ;
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 2/15 userfaultfd: introduce struct mfill_state
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic() passes a lot of parameters down to its callees.
Aggregate them all into mfill_state structure and pass this structure to functions that implement various UFFDIO_ commands.
Tracking the state in a structure will allow moving the code that retries copying of data for UFFDIO_COPY into mfill_atomic_pte_copy() and make the loop in mfill_atomic() identical for all UFFDIO operations on PTE-mapped memory.
The mfill_state definition is deliberately local to mm/userfaultfd.c, hence shmem_mfill_atomic_pte() is not updated.
[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap() WARN]
--- mm/userfaultfd.c | 147 ++++++++++++++++++++++++++--------------------- 1 file changed, 81 insertions(+), 66 deletions(-)
mm/userfaultfd.c +81 -66
@@ -20,6 +20,20 @@#include "internal.h" #include "swap.h" +struct mfill_state { +struct userfaultfd_ctx *ctx; +unsigned long src_start; +unsigned long dst_start; +unsigned long len; +uffd_flags_t flags; + +struct vm_area_struct *vma; +unsigned long src_addr; +unsigned long dst_addr; +struct folio *folio; +pmd_t *pmd; +}; +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {
@@ -272,17 +286,17 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_atomic_pte_copy (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static int mfill_atomic_pte_copy (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +unsigned long src_addr = state->src_addr; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio; +int ret; -if (!*foliop) { +if (!state->folio) { ret = -ENOMEM ; folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, dst_addr);
@@ -294,13 +308,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,/* fallback to copy_from_user outside mmap_lock */ if (unlikely (ret)) { ret = -ENOENT ; - *foliop = folio; + state->folio = folio;/* don't free the page */ goto out; } }else { - folio = *foliop; - *foliop =NULL ; + folio = state->folio; + state->folio =NULL ; }/*
@@ -357,10 +371,11 @@ static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,return ret; } -static int mfill_atomic_pte_zeropage (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr) +static int mfill_atomic_pte_zeropage (struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl;int ret;
@@ -392,13 +407,14 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, }/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ -static int mfill_atomic_pte_continue (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_continue (struct mfill_state *state) { -struct inode *inode =file_inode (dst_vma->vm_file); +struct vm_area_struct *dst_vma = state->vma; +unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct inode *inode =file_inode (dst_vma->vm_file); +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;struct folio *folio;struct page *page;int ret;
@@ -436,15 +452,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ -static int mfill_atomic_pte_poison (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -uffd_flags_t flags) +static int mfill_atomic_pte_poison (struct mfill_state *state) { -int ret; +struct vm_area_struct *dst_vma = state->vma;struct mm_struct *dst_mm = dst_vma->vm_mm; +unsigned long dst_addr = state->dst_addr; +pmd_t *dst_pmd = state->pmd;pte_t _dst_pte, *dst_pte;spinlock_t *ptl; +int ret; _dst_pte =make_pte_marker (PTE_MARKER_POISONED ); ret = -EAGAIN ;
@@ -668,22 +684,20 @@ extern ssize_t mfill_atomic_hugetlb(struct userfaultfd_ctx *ctx, uffd_flags_t flags);#endif /* CONFIG_HUGETLB_PAGE */ -static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop) +static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state) { +struct vm_area_struct *dst_vma = state->vma; +unsigned long src_addr = state->src_addr; +unsigned long dst_addr = state->dst_addr; +struct folio **foliop = &state->folio; +uffd_flags_t flags = state->flags; +pmd_t *dst_pmd = state->pmd;ssize_t err; -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { -return mfill_atomic_pte_continue (dst_pmd, dst_vma, - dst_addr, flags); - }else if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) { -return mfill_atomic_pte_poison (dst_pmd, dst_vma, - dst_addr, flags); - } +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) +return mfill_atomic_pte_continue (state); +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_POISON )) +return mfill_atomic_pte_poison (state);/* * The normal page fault path for a shmem will invoke the
@@ -697,12 +711,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,*/ if (!(dst_vma->vm_flags &VM_SHARED )) {if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) - err =mfill_atomic_pte_copy (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); + err =mfill_atomic_pte_copy (state);else - err =mfill_atomic_pte_zeropage (dst_pmd, - dst_vma, dst_addr); + err =mfill_atomic_pte_zeropage (state); }else { err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, src_addr,
@@ -718,13 +729,20 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long len, uffd_flags_t flags) { +struct mfill_state state = (struct mfill_state ){ + .ctx = ctx, + .dst_start = dst_start, + .src_start = src_start, + .flags = flags, + .len = len, + .src_addr = src_start, + .dst_addr = dst_start, + };struct mm_struct *dst_mm = ctx->mm;struct vm_area_struct *dst_vma; +long copied =0 ;ssize_t err;pmd_t *dst_pmd; -unsigned long src_addr, dst_addr; -long copied; -struct folio *folio;/* * Sanitize the command parameters:
@@ -736,10 +754,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); - src_addr = src_start; - dst_addr = dst_start; - copied =0 ; - folio =NULL ; retry:/* * Make sure the vma is not shared, that the dst range is
@@ -750,6 +764,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =PTR_ERR (dst_vma); goto out; } + state.vma = dst_vma;/* * If memory mappings are changing because of non-cooperative
@@ -790,12 +805,12 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) goto out_unlock; -while (src_addr < src_start + len) { -pmd_t dst_pmdval; +while (state.src_addr < src_start + len) { +VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -VM_WARN_ON_ONCE (dst_addr >= dst_start + len); +pmd_t dst_pmdval; - dst_pmd =mm_alloc_pmd (dst_mm, dst_addr); + dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr);if (unlikely (!dst_pmd)) { err = -ENOMEM ;break ;
@@ -827,34 +842,34 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,* tables under us; pte_offset_map_lock() will deal with that. */ - err =mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, - src_addr, flags, &folio); + state.pmd = dst_pmd; + err =mfill_atomic_pte (&state);cond_resched ();if (unlikely (err == -ENOENT )) {void *kaddr;up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); -VM_WARN_ON_ONCE (!folio); +uffd_mfill_unlock (state.vma); +VM_WARN_ON_ONCE (!state.folio); - kaddr =kmap_local_folio (folio,0 ); + kaddr =kmap_local_folio (state.folio,0 ); err =copy_from_user (kaddr, - (const void __user *) src_addr, + (const void __user *)state.src_addr,PAGE_SIZE );kunmap_local (kaddr);if (unlikely (err)) { err = -EFAULT ; goto out; } -flush_dcache_folio (folio); +flush_dcache_folio (state.folio); goto retry; }else -VM_WARN_ON_ONCE (folio); +VM_WARN_ON_ONCE (state.folio);if (!err) { - dst_addr +=PAGE_SIZE ; - src_addr +=PAGE_SIZE ; + state.dst_addr +=PAGE_SIZE ; + state.src_addr +=PAGE_SIZE ; copied +=PAGE_SIZE ;if (fatal_signal_pending (current))
@@ -866,10 +881,10 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, out_unlock:up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); +uffd_mfill_unlock (state.vma); out: -if (folio) -folio_put (folio); +if (state.folio) +folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.53.0
Re: [PATCH v4 02/15] userfaultfd: introduce struct mfill_state
Quoted 15 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
mfill_atomic()passes a lot of parameters down to its callees.Aggregate them all into
mfill_statestructure and pass this structure to functions that implement variousUFFDIO_commands.Tracking the state in a structure will allow moving the code that retries copying of data for
UFFDIO_COPYintomfill_atomic_pte_copy()and make the loop inmfill_atomic()identical for all UFFDIO operations on PTE-mapped memory.The
mfill_statedefinition is deliberately local tomm/userfaultfd.c, henceshmem_mfill_atomic_pte()is not updated.
Quoted 6 lines
[[email protected]: properly initialize mfill_state.len to fix
folio_add_new_anon_rmap()WARN]
Looks good to me,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerPATCH v4 3/15 userfaultfd: introduce mfill_establish_pmd() helper
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
There is a lengthy code chunk in mfill_atomic() that establishes the PMD for UFFDIO operations. This code may be called twice: first time when the copy is performed with VMA/mm locks held and the other time after the copy is retried with locks dropped.
Move the code that establishes a PMD into a helper function so it can be reused later during refactoring of mfill_atomic_pte_copy().
--- mm/userfaultfd.c | 102 ++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 50 deletions(-)
mm/userfaultfd.c +52 -50
@@ -157,6 +157,56 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) +{ +pgd_t *pgd; +p4d_t *p4d; +pud_t *pud; + + pgd =pgd_offset (mm, address); + p4d =p4d_alloc (mm, pgd, address); +if (!p4d) +return NULL ; + pud =pud_alloc (mm, p4d, address); +if (!pud) +return NULL ; +/* +* Note that we didn't run this because the pmd was +* missing, the *pmd may be already established and in +* turn it may also be a trans_huge_pmd. +*/ +return pmd_alloc (mm, pud, address); +} + +static int mfill_establish_pmd (struct mfill_state *state) +{ +struct mm_struct *dst_mm = state->ctx->mm; +pmd_t *dst_pmd, dst_pmdval; + + dst_pmd =mm_alloc_pmd (dst_mm, state->dst_addr); +if (unlikely (!dst_pmd)) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +if (unlikely (pmd_none (dst_pmdval)) && +unlikely (__pte_alloc (dst_mm, dst_pmd))) +return -ENOMEM ; + + dst_pmdval =pmdp_get_lockless (dst_pmd); +/* +* If the dst_pmd is THP don't override it and just be strict. +* (This includes the case where the PMD used to be THP and +* changed back to none after __pte_alloc().) +*/ +if (unlikely (!pmd_present (dst_pmdval) ||pmd_leaf (dst_pmdval))) +return -EEXIST ; +if (unlikely (pmd_bad (dst_pmdval))) +return -EFAULT ; + + state->pmd = dst_pmd; +return 0 ; +} +/* Check if dst_addr is outside of file's size. Must be called with ptl held. */ static bool mfill_file_over_size (struct vm_area_struct *dst_vma,unsigned long dst_addr)
@@ -489,27 +539,6 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)return ret; } -static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) -{ -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; - - pgd =pgd_offset (mm, address); - p4d =p4d_alloc (mm, pgd, address); -if (!p4d) -return NULL ; - pud =pud_alloc (mm, p4d, address); -if (!pud) -return NULL ; -/* -* Note that we didn't run this because the pmd was -* missing, the *pmd may be already established and in -* turn it may also be a trans_huge_pmd. -*/ -return pmd_alloc (mm, pud, address); -} -#ifdef CONFIG_HUGETLB_PAGE /* * mfill_atomic processing for HUGETLB vmas. Note that this routine is
@@ -742,7 +771,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err; -pmd_t *dst_pmd;/* * Sanitize the command parameters:
@@ -808,41 +836,15 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len); -pmd_t dst_pmdval; - - dst_pmd =mm_alloc_pmd (dst_mm, state.dst_addr); -if (unlikely (!dst_pmd)) { - err = -ENOMEM ; + err =mfill_establish_pmd (&state); +if (err)break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -if (unlikely (pmd_none (dst_pmdval)) && -unlikely (__pte_alloc (dst_mm, dst_pmd))) { - err = -ENOMEM ; -break ; - } - dst_pmdval =pmdp_get_lockless (dst_pmd); -/* -* If the dst_pmd is THP don't override it and just be strict. -* (This includes the case where the PMD used to be THP and -* changed back to none after __pte_alloc().) -*/ -if (unlikely (!pmd_present (dst_pmdval) || -pmd_trans_huge (dst_pmdval))) { - err = -EEXIST ; -break ; - } -if (unlikely (pmd_bad (dst_pmdval))) { - err = -EFAULT ; -break ; - }/* * For shmem mappings, khugepaged is allowed to remove page * tables under us; pte_offset_map_lock() will deal with that. */ - state.pmd = dst_pmd; err =mfill_atomic_pte (&state);cond_resched ();
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 4/15 userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from mfill_atomic() into a helper function.
This function will be used later during refactoring of mfill_atomic_pte_copy().
Add a counterpart mfill_put_vma() helper that unlocks the VMA and releases map_changing_lock.
--- mm/userfaultfd.c | 125 ++++++++++++++++++++++++++++------------------- 1 file changed, 75 insertions(+), 50 deletions(-)
mm/userfaultfd.c +75 -50
@@ -157,6 +157,75 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) }#endif +static void mfill_put_vma (struct mfill_state *state) +{ +if (!state->vma) +return ; + +up_read (&state->ctx->map_changing_lock); +uffd_mfill_unlock (state->vma); + state->vma =NULL ; +} + +static int mfill_get_vma (struct mfill_state *state) +{ +struct userfaultfd_ctx *ctx = state->ctx; +uffd_flags_t flags = state->flags; +struct vm_area_struct *dst_vma; +int err; + +/* +* Make sure the vma is not shared, that the dst range is +* both valid and fully within a single existing vma. +*/ + dst_vma =uffd_mfill_lock (ctx->mm, state->dst_start, state->len); +if (IS_ERR (dst_vma)) +return PTR_ERR (dst_vma); + +/* +* If memory mappings are changing because of non-cooperative +* operation (e.g. mremap) running in parallel, bail out and +* request the user to retry later +*/ +down_read (&ctx->map_changing_lock); + state->vma = dst_vma; + err = -EAGAIN ; +if (atomic_read (&ctx->mmap_changing)) + goto out_unlock; + + err = -EINVAL ; + +/* +* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but +* it will overwrite vm_ops, so vma_is_anonymous must return false. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && + dst_vma->vm_flags &VM_SHARED )) + goto out_unlock; + +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ +if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) + goto out_unlock; + +if (is_vm_hugetlb_page (dst_vma)) +return 0 ; + +if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + goto out_unlock; +if (!vma_is_shmem (dst_vma) && +uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + goto out_unlock; + +return 0 ; + +out_unlock: +mfill_put_vma (state); +return err; +} +static pmd_t *mm_alloc_pmd (struct mm_struct *mm,unsigned long address) {pgd_t *pgd;
@@ -767,8 +836,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, .src_addr = src_start, .dst_addr = dst_start, }; -struct mm_struct *dst_mm = ctx->mm; -struct vm_area_struct *dst_vma;long copied =0 ;ssize_t err;
@@ -783,56 +850,17 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (dst_start + len <= dst_start); retry: -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ - dst_vma =uffd_mfill_lock (dst_mm, dst_start, len); -if (IS_ERR (dst_vma)) { - err =PTR_ERR (dst_vma); + err =mfill_get_vma (&state); +if (err) goto out; - } - state.vma = dst_vma; - -/* -* If memory mappings are changing because of non-cooperative -* operation (e.g. mremap) running in parallel, bail out and -* request the user to retry later -*/ -down_read (&ctx->map_changing_lock); - err = -EAGAIN ; -if (atomic_read (&ctx->mmap_changing)) - goto out_unlock; - - err = -EINVAL ; -/* -* shmem_zero_setup is invoked in mmap for MAP_ANONYMOUS|MAP_SHARED but -* it will overwrite vm_ops, so vma_is_anonymous must return false. -*/ -if (WARN_ON_ONCE (vma_is_anonymous (dst_vma) && - dst_vma->vm_flags &VM_SHARED )) - goto out_unlock; - -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) - goto out_unlock;/* * If this is a HUGETLB vma, pass off to appropriate routine */ -if (is_vm_hugetlb_page (dst_vma)) -return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, +if (is_vm_hugetlb_page (state.vma)) +return mfill_atomic_hugetlb (ctx, state.vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; -while (state.src_addr < src_start + len) {VM_WARN_ON_ONCE (state.dst_addr >= dst_start + len);
@@ -851,8 +879,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,if (unlikely (err == -ENOENT )) {void *kaddr; -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state);VM_WARN_ON_ONCE (!state.folio); kaddr =kmap_local_folio (state.folio,0 );
@@ -881,9 +908,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,break ; } -out_unlock: -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (state.vma); +mfill_put_vma (&state); out:if (state.folio)folio_put (state.folio);
Signature · 1 line
2.53.0
Re: [PATCH v4 04/15] userfaultfd: introduce mfill_get_vma() and mfill_put_vma()
Quoted 8 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Split the code that finds, locks and verifies VMA from
mfill_atomic()into a helper function.This function will be used later during refactoring of
mfill_atomic_pte_copy().
Quoted 8 lines
Add a counterpart
mfill_put_vma()helper that unlocks the VMA and releasesmap_changing_lock.[[email protected]: fix lock leak in mfill_get_vma()]Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Andrei Vagin <[email protected]>---
Looks good to me,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerPATCH v4 5/15 userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of UFFDIO_COPY for anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either with mm_lock or with per-VMA lock).
In that case, mfill_atomic() releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.
Since this retry-reget dance is only relevant for UFFDIO_COPY and it never happens for other UFFDIO_ operations, make it a part of mfill_atomic_pte_copy() that actually implements UFFDIO_COPY for anonymous memory.
As a temporal safety measure to avoid breaking biscection mfill_atomic_pte_copy() makes sure to never return -ENOENT so that the loop in mfill_atomic() won't retry copiyng outside of mmap_lock. This is removed later when shmem implementation will be updated later and the loop in mfill_atomic() will be adjusted.
--- mm/userfaultfd.c | 75 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 24 deletions(-)
mm/userfaultfd.c +51 -24
@@ -405,35 +405,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } +static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +{ +unsigned long src_addr = state->src_addr; +void *kaddr; +int err; + +/* retry copying with mm_lock dropped */ +mfill_put_vma (state); + + kaddr =kmap_local_folio (folio,0 ); + err =copy_from_user (kaddr, (const void __user *) src_addr,PAGE_SIZE ); +kunmap_local (kaddr); +if (unlikely (err)) +return -EFAULT ; + +flush_dcache_folio (folio); + +/* reget VMA and PMD, they could change underneath us */ + err =mfill_get_vma (state); +if (err) +return err; + + err =mfill_establish_pmd (state); +if (err) +return err; + +return 0 ; +} +static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out; - } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) { +/* +* Fallback to copy_from_user outside mmap_lock. +* If retry is successful, mfill_copy_folio_locked() returns +* with locks retaken by mfill_get_vma(). +* If there was an error, we must mfill_put_vma() anyway and it +* will take care of unlocking if needed. +*/ + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto out_release; }/*
@@ -443,17 +471,16 @@ static int mfill_atomic_pte_copy(struct mfill_state *state)*/ __folio_mark_uptodate (folio); - ret = -ENOMEM ; -if (mem_cgroup_charge (folio, dst_vma->vm_mm,GFP_KERNEL )) - goto out_release; - - ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + ret =mfill_atomic_install_pte (state->pmd, state->vma, dst_addr, &folio->page, true, flags);if (ret) goto out_release; out:return ret; out_release: +/* Don't return -ENOENT so that our caller won't retry */ +if (ret == -ENOENT ) + ret = -EFAULT ;folio_put (folio); goto out; }
Signature · 1 line
2.53.0
Re: [PATCH v4 05/15] userfaultfd: retry copying with locks dropped in mfill_atomic_pte_copy()
Quoted 21 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Implementation of
UFFDIO_COPYfor anonymous memory might fail to copy data from userspace buffer when the destination VMA is locked (either withmm_lockor with per-VMA lock).In that case,
mfill_atomic()releases the locks, retries copying the data with locks dropped and then re-locks the destination VMA and re-establishes PMD.Since this retry-reget dance is only relevant for
UFFDIO_COPYand it never happens for otherUFFDIO_operations, make it a part ofmfill_atomic_pte_copy()that actually implementsUFFDIO_COPYfor anonymous memory.As a temporal safety measure to avoid breaking biscection
mfill_atomic_pte_copy()makes sure to never return-ENOENTso that the loop inmfill_atomic()won't retry copiyng outside ofmmap_lock. This is removed later when shmem implementation will be updated later and the loop inmfill_atomic()will be adjusted.
Quoted 6 lines
[[email protected]: update mfill_copy_folio_retry()]Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Andrew Morton <[email protected]>---
Looks good to me,
Quoted 39 lines
mm/userfaultfd.c | 75 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 24 deletions(-)mm/userfaultfd.c +6
-15@@ -405,35 +405,63 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)static int mfill_atomic_pte_copy (struct mfill_state *state) { -struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;unsigned long src_addr = state->src_addr;uffd_flags_t flags = state->flags; -pmd_t *dst_pmd = state->pmd;struct folio *folio;int ret; -if (!state->folio) { - ret = -ENOMEM ; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , dst_vma, - dst_addr); -if (!folio) - goto out; + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , state->vma, dst_addr); +if (!folio) +return -ENOMEM ; - ret =mfill_copy_folio_locked (folio, src_addr); + ret = -ENOMEM ; +if (mem_cgroup_charge (folio, state->vma->vm_mm,GFP_KERNEL )) + goto out_release; -/* fallback to copy_from_user outside mmap_lock */ -if (unlikely (ret)) { - ret = -ENOENT ; - state->folio = folio; -/* don't free the page */ - goto out;
mm/userfaultfd.c static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) · line 426
Quoted 10 lines
mm/userfaultfd.c +6
-4- } - }else { - folio = state->folio; - state->folio =NULL ; + ret =mfill_copy_folio_locked (folio, src_addr); +if (unlikely (ret)) { +/* +* Fallback to copy_from_user outside mmap_lock. +* If retry is successful, mfill_copy_folio_locked() returns +* with locks retaken by mfill_get_vma().
nit: mfill_copy_folio_locked() -> mfill_copy_folio_retry();
Quoted 9 lines
mm/userfaultfd.c +6
-0+* If there was an error, we must mfill_put_vma() anyway and it +* will take care of unlocking if needed. +*/ + ret =mfill_copy_folio_retry (state, folio); +if (ret) + goto out_release; }/*
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mike Rapoport
maintainerPATCH v4 6/15 userfaultfd: move vma_can_userfault out of line
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault() has grown pretty big and it's not called on performance critical path.
Move it out of line.
No functional changes.
--- include/linux/userfaultfd_k.h | 35 ++--------------------------------- mm/userfaultfd.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 33 deletions(-)
include/linux/userfaultfd_k.h +2 -33
@@ -208,39 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags, -bool wp_async) -{ - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true; - -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && - !vma_is_anonymous (vma)) -return false; - -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async);static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {
mm/userfaultfd.c +33 -0
@@ -2018,6 +2018,39 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,return moved ? moved : err; } +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async) +{ + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + +if ((vm_flags &VM_UFFD_MINOR ) && + (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) +return false; + +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem & hugetlbfs are not supported but only +* anonymous. +*/ +if (!uffd_supports_wp_marker () && (vm_flags &VM_UFFD_WP ) && + !vma_is_anonymous (vma)) +return false; + +/* By default, allow any of anon|shmem|hugetlb */ +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +static void userfaultfd_set_vm_flags (struct vm_area_struct *vma,vm_flags_t vm_flags) {
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 7/15 userfaultfd: introduce vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Current userfaultfd implementation works only with memory managed by core
MM: anonymous, shmem and hugetlb.
First, there is no fundamental reason to limit userfaultfd support only to the core memory types and userfaults can be handled similarly to regular page faults provided a VMA owner implements appropriate callbacks.
Second, historically various code paths were conditioned on vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of these conditions can be expressed as operations implemented by a particular memory type.
Introduce vm_uffd_ops extension to vm_operations_struct that will delegate memory type specific operations to a VMA owner.
Operations for anonymous memory are handled internally in userfaultfd using anon_uffd_ops that implicitly assigned to anonymous VMAs.
Start with a single operation, ->can_userfault() that will verify that a VMA meets requirements for userfaultfd support at registration time.
Implement that method for anonymous, shmem and hugetlb and move relevant parts of vma_can_userfault() into the new callbacks.
--- include/linux/mm.h | 5 ++++ include/linux/userfaultfd_k.h | 6 +++++ mm/hugetlb.c | 15 ++++++++++++ mm/shmem.c | 15 ++++++++++++ mm/userfaultfd.c | 44 ++++++++++++++++++++++++----------- 5 files changed, 72 insertions(+), 13 deletions(-)
include/linux/mm.h +5 -0
@@ -741,6 +741,8 @@ struct vm_fault {*/ }; +struct vm_uffd_ops ; +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -826,6 +828,9 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +const struct vm_uffd_ops *uffd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +6 -0
@@ -80,6 +80,12 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); +/* VMA userfaultfd operations */ +struct vm_uffd_ops { +/* Checks if a VMA can support userfaultfd */ +bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +}; +/* A combined operation mode + behavior flags. */ typedef unsigned int __bitwise uffd_flags_t ;
mm/hugetlb.c +15 -0
@@ -4818,6 +4818,18 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_USERFAULTFD +static bool hugetlb_can_userfault (struct vm_area_struct *vma, +vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops hugetlb_uffd_ops = { + .can_userfault = hugetlb_can_userfault, +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -4831,6 +4843,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
mm/shmem.c +15 -0
@@ -3294,6 +3294,15 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,shmem_inode_unacct_blocks (inode,1 ); return ret; } + +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +return true; +} + +static const struct vm_uffd_ops shmem_uffd_ops = { + .can_userfault = shmem_can_userfault, +};#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS
@@ -5313,6 +5322,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5322,6 +5334,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .uffd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)
mm/userfaultfd.c +31 -13
71 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH v4 07/15] userfaultfd: introduce vm_uffd_ops
Hi Mike,
Quoted 20 lines · Mike Rapoport
mm/userfaultfd.c +3
-8@@ -2021,34 +2040,33 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags,bool wp_async) { - vm_flags &= __VM_UFFD_FLAGS; +const struct vm_uffd_ops *ops =vma_uffd_ops (vma); -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; + vm_flags &= __VM_UFFD_FLAGS;/* -* If wp async enabled, and WP is the only mode enabled, allow any +* If WP is the only mode enabled and context is wp async, allow any * memory type. */
mm/userfaultfd.c ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2060
Quoted 10 lines
mm/userfaultfd.c +7
-0if (wp_async && (vm_flags ==VM_UFFD_WP ))return true; +/* For any other mode reject VMAs that don't implement vm_uffd_ops */ +if (!ops) +return false; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; +
Doesn't moving the VM_DROPPABLE check after the wp_async early return make the issue David raised in [1] possible again (i.e. moving swapbacked pages into a droppable region using UFFDIO_MOVE)? With this change, VM_DROPPABLE pages can now be registered with uffd under wp_async, which allows UFFDIO_MOVE.
I think the VM_DROPPABLE check should go right before the wp_async check. And it would probably benefit from a comment...
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Tal
Mike Rapoport
maintainerRe: [PATCH v4 07/15] userfaultfd: introduce vm_uffd_ops
Quoted 40 lines · Tal Zussman
Hi Mike,
Quoted 20 lines · Mike Rapoport
(quoted diff) +3
-8@@ -2021,34 +2040,33 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, bool wp_async) { - vm_flags &= __VM_UFFD_FLAGS; + const struct vm_uffd_ops *ops = vma_uffd_ops(vma); - if (vma->vm_flags & VM_DROPPABLE) - return false; - - if ((vm_flags & VM_UFFD_MINOR) && - (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma))) - return false; + vm_flags &= __VM_UFFD_FLAGS; /* - * If wp async enabled, and WP is the only mode enabled, allow any + * If WP is the only mode enabled and context is wp async, allow any * memory type. */Review ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2060
Quoted 10 lines
(quoted diff) +7
-0if (wp_async && (vm_flags == VM_UFFD_WP)) return true; + /* For any other mode reject VMAs that don't implement vm_uffd_ops */ + if (!ops) + return false; + + if (vma->vm_flags & VM_DROPPABLE) + return false; +Doesn't moving the
VM_DROPPABLEcheck after thewp_asyncearly return make the issue David raised in [1] possible again (i.e. moving swapbacked pages into a droppable region usingUFFDIO_MOVE)? With this change,VM_DROPPABLEpages can now be registered with uffd underwp_async, which allowsUFFDIO_MOVE.
Quoted 2 lines
I think the
VM_DROPPABLEcheck should go right before thewp_asynccheck. And it would probably benefit from a comment...
You are right.
Andrew, if it's not too late can you please add this fixup:
mm/userfaultfd.c +3 -3
@@ -2019,6 +2019,9 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, {const struct vm_uffd_ops *ops =vma_uffd_ops (vma); +if (vma->vm_flags &VM_DROPPABLE ) +return false; + vm_flags &= __VM_UFFD_FLAGS;/*
@@ -2032,9 +2035,6 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,if (!ops)return false; -if (vma->vm_flags &VM_DROPPABLE ) -return false; -/* * If user requested uffd-wp but not enabled pte markers for * uffd-wp, then only anonymous memory is supported
Quoted 2 lines
Thanks,
Tal
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v4 8/15 shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.
Instead of using shmem_get_folio() for that, add a get_folio_noalloc() method to 'struct vm_uffd_ops' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Implement get_folio_noalloc() method for shmem and slightly refactor userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support this new API.
--- include/linux/userfaultfd_k.h | 7 +++++++ mm/shmem.c | 15 ++++++++++++++- mm/userfaultfd.c | 34 ++++++++++++++++++---------------- 3 files changed, 39 insertions(+), 17 deletions(-)
include/linux/userfaultfd_k.h +7 -0
@@ -84,6 +84,13 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);struct vm_uffd_ops {/* Checks if a VMA can support userfaultfd */ bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* +* Called to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +* The returned folio is locked and with reference held. +*/ +struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); };/* A combined operation mode + behavior flags. */
mm/shmem.c +14 -1
@@ -3295,13 +3295,26 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, return ret; } +static struct folio *shmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +} +static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) {return true; }static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .can_userfault = shmem_can_userfault, + .get_folio_noalloc = shmem_get_folio_noalloc, };#endif /* CONFIG_USERFAULTFD */
mm/userfaultfd.c +18 -16
@@ -191,6 +191,7 @@ static int mfill_get_vma(struct mfill_state *state)struct userfaultfd_ctx *ctx = state->ctx;uffd_flags_t flags = state->flags;struct vm_area_struct *dst_vma; +const struct vm_uffd_ops *ops;int err;/*
@@ -232,10 +233,12 @@ static int mfill_get_vma(struct mfill_state *state)if (is_vm_hugetlb_page (dst_vma))return 0 ; -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) + ops =vma_uffd_ops (dst_vma); +if (!ops) goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && + !ops->get_folio_noalloc) goto out_unlock;return 0 ;
@@ -575,6 +578,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)static int mfill_atomic_pte_continue (struct mfill_state *state) {struct vm_area_struct *dst_vma = state->vma; +const struct vm_uffd_ops *ops =vma_uffd_ops (dst_vma);unsigned long dst_addr = state->dst_addr;pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct inode *inode =file_inode (dst_vma->vm_file);
@@ -584,17 +588,16 @@ static int mfill_atomic_pte_continue(struct mfill_state *state)struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); -/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; +if (!ops) { +VM_WARN_ONCE (1 ,"UFFDIO_CONTINUE for unsupported VMA" ); +return -EOPNOTSUPP ; } + folio = ops->get_folio_noalloc (inode, pgoff); +/* Our caller expects us to return -EFAULT if we failed to find folio */ +if (IS_ERR_OR_NULL (folio)) +return -EFAULT ; + page =folio_file_page (folio, pgoff);if (PageHWPoison (page)) { ret = -EIO ;
@@ -607,13 +610,12 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 9/15 userfaultfd: introduce vm_uffd_ops->alloc_folio()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
and use it to refactor mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy().
mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio
Split a __mfill_atomic_pte() helper that handles both cases and uses newly introduced vm_uffd_ops->alloc_folio() to allocate the folio.
Pass the ops structure from the callers to __mfill_atomic_pte() to later allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.
Note, that the new ops method is called alloc_folio() rather than folio_alloc() to avoid clash with alloc_tag macro folio_alloc().
--- include/linux/userfaultfd_k.h | 6 +++ mm/userfaultfd.c | 92 ++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 44 deletions(-)
include/linux/userfaultfd_k.h +6 -0
@@ -91,6 +91,12 @@ struct vm_uffd_ops {* The returned folio is locked and with reference held. */ struct folio *(*get_folio_noalloc)(struct inode *inode,pgoff_t pgoff); +/* +* Called during resolution of UFFDIO_COPY request. +* Should allocate and return a folio or NULL if allocation fails. +*/ +struct folio *(*alloc_folio)(struct vm_area_struct *vma, +unsigned long addr); };/* A combined operation mode + behavior flags. */
mm/userfaultfd.c +48 -44
142 lines not on this page · raw message
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 10/15 shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them in __mfill_atomic_pte() to add shmem folios to page cache and remove them in case of error.
Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and drop shmem_mfill_atomic_pte().
Since userfaultfd now does not reference any functions from shmem, drop include if linux/shmem_fs.h from mm/userfaultfd.c
mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd, make it static.
--- include/linux/shmem_fs.h | 14 ---- include/linux/userfaultfd_k.h | 19 +++-- mm/shmem.c | 148 ++++++++++++---------------------- mm/userfaultfd.c | 80 +++++++++--------- 4 files changed, 106 insertions(+), 155 deletions(-)
include/linux/shmem_fs.h +0 -14
@@ -221,20 +221,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +14 -5
@@ -97,6 +97,20 @@ struct vm_uffd_ops {*/ struct folio *(*alloc_folio)(struct vm_area_struct *vma,unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request. +* Should only be called with a folio returned by alloc_folio() above. +* The folio will be set to locked. +* Returns 0 on success, error code on failure. +*/ +int (*filemap_add)(struct folio *folio,struct vm_area_struct *vma, +unsigned long addr); +/* +* Called during resolution of UFFDIO_COPY request on the error +* handling path. +* Should revert the operation of ->filemap_add(). +*/ +void (*filemap_remove)(struct folio *folio,struct vm_area_struct *vma); };/* A combined operation mode + behavior flags. */
@@ -130,11 +144,6 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at/* Flags controlling behavior. These behavior changes are mode-independent. */ #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0) -extern int mfill_atomic_install_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr,struct page *page, -bool newly_allocated,uffd_flags_t flags); -extern ssize_t mfill_atomic_copy (struct userfaultfd_ctx *ctx,unsigned long dst_start,unsigned long src_start,unsigned long len,uffd_flags_t flags);
mm/shmem.c +53 -95
177 lines not on this page · raw message
mm/userfaultfd.c +39 -41
149 lines not on this page · raw message
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 11/15 userfaultfd: mfill_atomic(): remove retry logic
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().
Drop the retry logic from mfill_atomic().
--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)
mm/userfaultfd.c +0 -24
@@ -29,7 +29,6 @@ struct mfill_state {struct vm_area_struct *vma;unsigned long src_addr;unsigned long dst_addr; -struct folio *folio;pmd_t *pmd; };
@@ -899,7 +898,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); -retry: err =mfill_get_vma (&state);if (err) goto out;
@@ -926,26 +924,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, err =mfill_atomic_pte (&state);cond_resched (); -if (unlikely (err == -ENOENT )) { -void *kaddr; - -mfill_put_vma (&state); -VM_WARN_ON_ONCE (!state.folio); - - kaddr =kmap_local_folio (state.folio,0 ); - err =copy_from_user (kaddr, - (const void __user *)state.src_addr, -PAGE_SIZE ); -kunmap_local (kaddr); -if (unlikely (err)) { - err = -EFAULT ; - goto out; - } -flush_dcache_folio (state.folio); - goto retry; - }else -VM_WARN_ON_ONCE (state.folio); -if (!err) { state.dst_addr +=PAGE_SIZE ; state.src_addr +=PAGE_SIZE ;
@@ -960,8 +938,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,mfill_put_vma (&state); out: -if (state.folio) -folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
Signature · 1 line
2.53.0
Mike Rapoport
maintainerRe: [PATCH v4 11/15] userfaultfd: mfill_atomic(): remove retry logic
Quoted 6 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Since
__mfill_atomic_pte()handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop inmfill_atomic().
Quoted 6 lines
Drop the retry logic from
mfill_atomic().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)
After discussion with David Carlier about potential replacement of VMA in mfill_copy_folio_retry(), I looked again in the code and realized that after all the rebases I didn't remove the bit that temporarily prevented returning ENOENT from __mfill_atomic_pte().
Andrew, can you please fold this into "userfaultfd: mfill_atomic(): remove retry logic"?
For a change it applies cleanly :)
commit 5173c8f4fd32f314907b3804217ef57d4e3a2220
Author: Mike Rapoport (Microsoft) <[email protected]> Date: Thu Apr 2 16:38:39 2026 +0300
userfaultfd: remove safety mesaure of not returning ENOENT from _copy
mm/userfaultfd.c +0 -3
@@ -530,9 +530,6 @@ static int __mfill_atomic_pte(struct mfill_state *state, ops->filemap_remove (folio, state->vma); err_folio_put:folio_put (folio); -/* Don't return -ENOENT so that our caller won't retry */ -if (ret == -ENOENT ) - ret = -EFAULT ;return ret; }
Signature · 2 lines
Sincerely yours, Mike.
Andrew Morton
maintainerRe: [PATCH v4 11/15] userfaultfd: mfill_atomic(): remove retry logic
Quoted 12 lines · On Thu, 2 Apr 2026 16:47:28 +0300 Mike Rapoport <[email protected]> wrote:
Quoted 6 lines
Drop the retry logic from
mfill_atomic().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/userfaultfd.c | 24 ------------------------ 1 file changed, 24 deletions(-)After discussion with David Carlier about potential replacement of VMA in
mfill_copy_folio_retry(), I looked again in the code and realized that after all the rebases I didn't remove the bit that temporarily prevented returning ENOENT from__mfill_atomic_pte().
Quoted 4 lines
Andrew, can you please fold this into "userfaultfd:
mfill_atomic(): remove retry logic"?For a change it applies cleanly :)
done.
Quoted 5 lines
commit
5173c8f4fd32f314907b3804217ef57d4e3a2220Author: Mike Rapoport (Microsoft) <[email protected]> Date: Thu Apr 2 16:38:39 2026 +0300userfaultfd: remove safety mesaure of not returning ENOENT from
_copy
s/mesaure/measure/
Was "_copy" intended?
Quoted 2 lines
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
I instafolded this. End result:
From: "Mike Rapoport (Microsoft)" <[email protected]>
Subject: userfaultfd: mfill_atomic(): remove retry logic
Date: Thu, 2 Apr 2026 07:11:52 +0300
Since __mfill_atomic_pte() handles the retry for both anonymous and shmem, there is no need to retry copying the date from the userspace in the loop in mfill_atomic().
Drop the retry logic from mfill_atomic().
---
mm/userfaultfd.c | 27 --------------------------- 1 file changed, 27 deletions(-)
mm/userfaultfd.c renamed from mm/userfaultfd.c~userfaultfd-mfill_atomic-remove-retry-logic +0 -27
@@ -29,7 +29,6 @@ struct mfill_state {struct vm_area_struct *vma;unsigned long src_addr;unsigned long dst_addr; -struct folio *folio;pmd_t *pmd; };
@@ -531,9 +530,6 @@ err_filemap_remove: ops->filemap_remove (folio, state->vma); err_folio_put:folio_put (folio); -/* Don't return -ENOENT so that our caller won't retry */ -if (ret == -ENOENT ) - ret = -EFAULT ;return ret; }
@@ -899,7 +895,6 @@ static __always_inline ssize_t mfill_atoVM_WARN_ON_ONCE (src_start + len <= src_start);VM_WARN_ON_ONCE (dst_start + len <= dst_start); -retry: err =mfill_get_vma (&state);if (err) goto out;
@@ -926,26 +921,6 @@ retry: err =mfill_atomic_pte (&state);cond_resched (); -if (unlikely (err == -ENOENT )) { -void *kaddr; - -mfill_put_vma (&state); -VM_WARN_ON_ONCE (!state.folio); - - kaddr =kmap_local_folio (state.folio,0 ); - err =copy_from_user (kaddr, - (const void __user *)state.src_addr, -PAGE_SIZE ); -kunmap_local (kaddr); -if (unlikely (err)) { - err = -EFAULT ; - goto out; - } -flush_dcache_folio (state.folio); - goto retry; - }else -VM_WARN_ON_ONCE (state.folio); -if (!err) { state.dst_addr +=PAGE_SIZE ; state.src_addr +=PAGE_SIZE ;
@@ -960,8 +935,6 @@ retry:mfill_put_vma (&state); out: -if (state.folio) -folio_put (state.folio);VM_WARN_ON_ONCE (copied <0 );VM_WARN_ON_ONCE (err >0 );VM_WARN_ON_ONCE (!copied && !err);
_
Mike Rapoport
maintainerPATCH v4 12/15 mm: generalize handling of userfaults in __do_fault()
In reply to Mike RapoportFrom: Peter Xu <[email protected]>
When a VMA is registered with userfaulfd, its ->fault() method should check if a folio exists in the page cache and call handle_userfault() with appropriate mode:
- VM_UFFD_MINOR if VMA is registered in minor mode and the folio exists
- VM_UFFD_MISSING if VMA is registered in missing mode and the folio does not exist
Instead of calling handle_userfault() directly from a specific ->fault() handler, call __do_userfault() helper from the generic __do_fault().
For VMAs registered with userfaultfd the new __do_userfault() helper will check if the folio is found in the page cache using vm_uffd_ops->get_folio_noalloc() and call handle_userfault() with the appropriate mode.
Make vm_uffd_ops->get_folio_noalloc() required method for non-anonymous VMAs mapped at PTE level.
--- mm/memory.c | 43 +++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 12 ------------ mm/userfaultfd.c | 9 +++++++++ 3 files changed, 52 insertions(+), 12 deletions(-)
mm/memory.c +43 -0
@@ -5329,6 +5329,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ; } +#ifdef CONFIG_USERFAULTFD +static vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode; +struct folio *folio; + +if (!(userfaultfd_missing (vma) ||userfaultfd_minor (vma))) +return 0 ; + + inode =file_inode (vma->vm_file); + folio = vma->vm_ops->uffd_ops->get_folio_noalloc (inode, vmf->pgoff); +if (!IS_ERR_OR_NULL (folio)) { +/* +* TODO: provide a flag for get_folio_noalloc() to avoid +* locking (or even the extra reference?) +*/ +folio_unlock (folio); +folio_put (folio); +if (userfaultfd_minor (vma)) +return handle_userfault (vmf,VM_UFFD_MINOR ); + }else { +if (userfaultfd_missing (vma)) +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + +return 0 ; +} +#else +static inline vm_fault_t __do_userfault (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +/* * The mmap_lock must have been held on entry, and may have been * released depending on flags and vma->vm_ops->fault() return value.
@@ -5361,6 +5396,14 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)return VM_FAULT_OOM ; } +/* +* If this is a userfault trap, process it in advance before +* triggering the genuine fault handler. +*/ + ret =__do_userfault (vmf); +if (ret) +return ret; + ret = vma->vm_ops->fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY |VM_FAULT_DONE_COW )))
mm/shmem.c +0 -12
@@ -2489,13 +2489,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm :NULL ; folio =filemap_get_entry (inode->i_mapping, index); -if (folio && vma &&userfaultfd_minor (vma)) { -if (!xa_is_value (folio)) -folio_put (folio); - *fault_type =handle_userfault (vmf,VM_UFFD_MINOR ); -return 0 ; - } -if (xa_is_value (folio)) { error =shmem_swapin_folio (inode, index, &folio, sgp, gfp, vma, fault_type);
@@ -2540,11 +2533,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* Fast cache lookup and swap lookup did not find it: allocate. */ -if (vma &&userfaultfd_missing (vma)) { - *fault_type =handle_userfault (vmf,VM_UFFD_MISSING ); -return 0 ; - } -/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */ orders =shmem_allowable_huge_orders (inode, vma, index, write_end, false);if (orders >0 ) {
mm/userfaultfd.c +9 -0
@@ -2046,6 +2046,15 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !vma_is_anonymous(vma))return false; +/* +* File backed VMAs (except HugeTLB) must implement +* ops->get_folio_noalloc() because it's required by __do_userfault() +* in page fault handling. +*/ +if (!vma_is_anonymous (vma) && !is_vm_hugetlb_page (vma) && + !ops->get_folio_noalloc) +return false; +return ops->can_userfault (vma, vm_flags); }
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 13/15 KVM: guest_memfd: implement userfaultfd operations
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit 7677f7fd8be7 ("userfaultfd: add minor fault registration mode").
To use the same mechanisms for VMs that use guest_memfd to map their memory, guest_memfd should support userfaultfd operations.
Add implementation of vm_uffd_ops to guest_memfd.
--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)
mm/filemap.c +1 -0
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache
virt/kvm/guest_memfd.c +82 -2
114 lines not on this page · raw message
Signature · 1 line
2.53.0
Sean Christopherson
maintainerRe: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 28 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in commit
7677f7fd8be7("userfaultfd: add minor fault registration mode").To use the same mechanisms for VMs that use
guest_memfdto map their memory,guest_memfdshould support userfaultfd operations.Add implementation of
vm_uffd_opstoguest_memfd.Signed-off-by Nikita Kalyazin <[email protected]>Co-developed-by Mike Rapoport (Microsoft) <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- mm/filemap.c | 1 + virt/kvm/guest_memfd.c | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)mm/filemap.c +0
-0
mm/filemap.c void filemap_remove_folio(struct folio *folio) · line 265
Quoted 4 lines
mm/filemap.c +1
-0@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );
This can be EXPORT_SYMBOL_FOR_KVM so that the symbol is exported if and only if KVM is built as a module.
Quoted 15 lines
mm/filemap.c +0
-0/* * page_cache_delete_batch - delete several folios from page cache virt/kvm/guest_memfd.c +1
-0@@ -7,6 +7,7 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h"
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 114
Quoted 7 lines
virt/kvm/guest_memfd.c +4
-0@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,return __kvm_gmem_prepare_folio (kvm, slot, index, folio); } +static struct folio *kvm_gmem_get_folio_noalloc (struct inode *inode,pgoff_t pgoff) +{ +return __filemap_get_folio (inode->i_mapping, pgoff, +FGP_LOCK |FGP_ACCESSED ,0 );
Note, this will conflict with commit 6dad5447c7bf ("KVM: guest_memfd: Don't set FGP_ACCESSED when getting folios") sitting in
https://github.com/kvm-x86/linux.git gmem
I think the resolution is to just end up with:
static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
{
return filemap_lock_folio(inode->i_mapping, pgoff);
}However, I think that'll be a moot point in the end (the conflict will be avoided). More below.
Quoted 20 lines
virt/kvm/guest_memfd.c +4
-2+} +/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest. @@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)* Fast-path: See if folio is already present in mapping to avoid * policy_lookup. */ - folio =__filemap_get_folio (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED ,0 ); + folio =kvm_gmem_get_folio_noalloc (inode, index);if (!IS_ERR (folio))return folio;@@ -457,12 +463,86 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, }#endif /* CONFIG_NUMA */ +#ifdef CONFIG_USERFAULTFD
virt/kvm/guest_memfd.c static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, · line 476
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+static bool kvm_gmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) +{ +struct inode *inode =file_inode (vma->vm_file); + +/* +* Only support userfaultfd for guest_memfd with INIT_SHARED flag. +* This ensures the memory can be mapped to userspace. +*/ +if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )) +return false;
I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.
Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.
My objection to an early, global "can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. for VM_UFFD_MISSING and VM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in __do_userfault(), not broadly when VM_UFFD_MINOR is added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path in kvm_gmem_fault_user_mapping().
I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the INIT_SHARED check to ->alloc_folio() and ->get_folio_noalloc(). And if we extract kvm_gmem_is_shared_mem() now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.
The bonus side effect is that dropping guest_memfd's more "complex" can_userfault means the only remaining check is constant based on the backing memory vs. the UFFD flags. If we want, the indirect call to a function can be replace with a constant vm_flags_t variable that enumerates the supported (or unsupported if we're feeling negative) flags, e.g.
include/linux/userfaultfd_k.h +2 -2
@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);/* VMA userfaultfd operations */ struct vm_uffd_ops { -/* Checks if a VMA can support userfaultfd */ -bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* What UFFD flags/modes are supported. */ +const vm_flags_t supported_uffd_flags;/* * Called to resolve UFFDIO_CONTINUE request. * Should return the folio found at pgoff in the VMA's pagecache if it
with usage like:
static const struct vm_uffd_ops shmem_uffd_ops = {
.supported_uffd_flags = __VM_UFFD_FLAGS,
.get_folio_noalloc = shmem_get_folio_noalloc,
.alloc_folio = shmem_mfill_folio_alloc,
.filemap_add = shmem_mfill_filemap_add,
.filemap_remove = shmem_mfill_filemap_remove,
};
Quoted 2 lines
virt/kvm/guest_memfd.c +2
-0+return true; +}
...
virt/kvm/guest_memfd.c
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+static const struct vm_uffd_ops kvm_gmem_uffd_ops = { + .can_userfault = kvm_gmem_can_userfault, + .get_folio_noalloc = kvm_gmem_get_folio_noalloc, + .alloc_folio = kvm_gmem_folio_alloc, + .filemap_add = kvm_gmem_filemap_add, + .filemap_remove = kvm_gmem_filemap_remove,
Please use kvm_gmem_uffd_xxx(). The names are a bit verbose, but these are waaay to generic of names as-is, e.g. kvm_gmem_folio_alloc() has implications and restrictions far beyond just allocating a folio.
All in all, somelike like so (completely untested):
--- include/linux/userfaultfd_k.h | 4 +- mm/filemap.c | 1 + mm/hugetlb.c | 8 +--- mm/shmem.c | 7 +-- mm/userfaultfd.c | 6 +-- virt/kvm/guest_memfd.c | 80 ++++++++++++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 19 deletions(-)
include/linux/userfaultfd_k.h +2 -2
@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);/* VMA userfaultfd operations */ struct vm_uffd_ops { -/* Checks if a VMA can support userfaultfd */ -bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* What UFFD flags/modes are supported. */ +const vm_flags_t supported_uffd_flags;/* * Called to resolve UFFDIO_CONTINUE request. * Should return the folio found at pgoff in the VMA's pagecache if it
mm/filemap.c +1 -0
@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES (filemap_remove_folio,"kvm" );/* * page_cache_delete_batch - delete several folios from page cache
mm/hugetlb.c +1 -7
@@ -4819,14 +4819,8 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf) }#ifdef CONFIG_USERFAULTFD -static bool hugetlb_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags) -{ -return true; -} -static const struct vm_uffd_ops hugetlb_uffd_ops = { - .can_userfault = hugetlb_can_userfault, + .supported_uffd_flags = __VM_UFFD_FLAGS, };#endif
mm/shmem.c +1 -6
@@ -3250,13 +3250,8 @@ static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)return folio; } -static bool shmem_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) -{ -return true; -} -static const struct vm_uffd_ops shmem_uffd_ops = { - .can_userfault = shmem_can_userfault, + .supported_uffd_flags = __VM_UFFD_FLAGS, .get_folio_noalloc = shmem_get_folio_noalloc, .alloc_folio = shmem_mfill_folio_alloc, .filemap_add = shmem_mfill_filemap_add,
mm/userfaultfd.c +3 -3
@@ -58,8 +58,8 @@ static struct folio *anon_alloc_folio(struct vm_area_struct *vma, }static const struct vm_uffd_ops anon_uffd_ops = { - .can_userfault = anon_can_userfault, - .alloc_folio = anon_alloc_folio, + .supported_uffd_flags = __VM_UFFD_FLAGS & ~VM_UFFD_MINOR , + .alloc_folio = anon_alloc_folio, };static const struct vm_uffd_ops *vma_uffd_ops (struct vm_area_struct *vma)
@@ -2055,7 +2055,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !ops->get_folio_noalloc)return false; -return ops->can_userfault (vma, vm_flags); +return ops->supported_uffd_flags & vm_flags; } static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
virt/kvm/guest_memfd.c +79 -1
110 lines not on this page · raw message
base-commit: d63beb006dba56d5fa219f106c7a97eb128c356f --
Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 112 lines · Sean Christopherson
Quoted 27 lines · Mike Rapoport
From: Nikita Kalyazin <[email protected]>
userfaultfd notifications about page faults used for live migration and snapshotting of VMs.
MISSING mode allows post-copy live migration and MINOR mode allows optimization for post-copy live migration for VMs backed with shared hugetlbfs or tmpfs mappings as described in detail in
commit7677f7fd8be7("userfaultfd: add minor fault registration
mode").To use the same mechanisms for VMs that use
guest_memfdto map their memory,guest_memfdshould support userfaultfd operations.Add implementation of
vm_uffd_opstoguest_memfd.Signed-off-by Nikita Kalyazin <[email protected]> Co-developed-by: Mike Rapoport (Microsoft) <[email protected]> Signed-off-by:
Mike Rapoport (Microsoft) <[email protected]> ---mm/filemap.c| 1 +virt/kvm/guest_memfd.c| 84 ++++++++++++
+++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-)mm/filemap.c index renamed from mm/filemap.c +0
-0
406cef06b684..a91582293118100644 ---a/mm/filemap.c+++ b/mm/filemap.c@@ -262,6 +262,7 @@ voidfilemap_remove_folio(struct folio *folio)Quoted 2 lines
filemap_free_folio(mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");This can be
EXPORT_SYMBOL_FOR_KVMso that the symbol is exported if and only if KVM is built as a module.Quoted 14 lines
/* *
page_cache_delete_batch- delete several folios from page
cache diff --gita/virt/kvm/guest_memfd.cb/virt/kvm/guest_memfd.cindex017d84a7adf3..46582feeed75100644 --- a/virt/kvm/guest_memfd.c+++b/virt/kvm/guest_memfd.c@@ -7,6 +7,7 @@ #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h>#include "
kvm_mm.h"mm/filemap.c index renamed from mm/filemap.c +1
-0@@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, return __kvm_gmem_prepare_folio(kvm, slot, index, folio); } +static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) +{ + return __filemap_get_folio(inode-Quoted 1 line
i_mapping, pgoff, +FGP_LOCK|Quoted 1 line
FGP_ACCESSED, 0);Note, this will conflict with commit
6dad5447c7bf("KVM:guest_memfd: Don't setFGP_ACCESSEDwhen getting folios") sitting inhttps://github.com/kvm-x86/linux.git gmem
I think the resolution is to just end up with:
static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode,
pgoff_tpgoff) { returnfilemap_lock_folio(inode->i_mapping, pgoff); }However, I think that'll be a moot point in the end (the conflict will be avoided). More below.
Quoted 14 lines
+} + /* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest. @@ -126,8 +133,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode,
pgoff_tindex) * Fast-
path: See if folio is already present in mapping to avoid *policy_lookup. */ - folio = __filemap_get_folio(inode-Quoted 1 line
i_mapping, index, -FGP_LOCK|
FGP_ACCESSED, 0); + folio = kvm_gmem_get_folio_noalloc(inode,
index); if (!IS_ERR(folio)) return folio;mm/filemap.c index renamed from mm/filemap.c +0
-0@@ -457,12 +463,86 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, } #endif /*
CONFIG_NUMA*/Quoted 7 lines
+#ifdef
CONFIG_USERFAULTFD+static bool kvm_gmem_can_userfault(structvm_area_struct*vma,vm_flags_tvm_flags) +{ + struct inode *inode =file_inode(vma->vm_file);
+ + /* + * Only support userfaultfd forguest_memfdwithINIT_SHAREDflag. + * This ensures the memory can be mapped
to userspace. + */ + if (!(GMEM_I(inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED)) + return false;I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.
Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.
My objection to an early, global "
can_userfault()" check is that
it's guaranteed to cause TOCTOU issues. E.g. forVM_UFFD_MISSING
andVM_UFFD_MINOR, the check on whether or not a given address can
be faulted in needs to happen in__do_userfault(), not broadly whenVM_UFFD_MINORis added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path inkvm_gmem_fault_user_mapping().I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the
INIT_SHAREDcheck to ->alloc_folio() and ->get_folio_noalloc(). And if we extractkvm_gmem_is_shared_mem()now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.
Quoted 5 lines
The bonus side effect is that dropping guest_memfd's more "complex"
can_userfaultmeans the only remaining check is constant based on the backing memory vs. the UFFD flags. If we want, the indirect
call to a function can be replace with a constantvm_flags_t
variable that enumerates the supported (or unsupported if we're
feeling negative) flags, e.g.
Thanks Sean. Checking for GUEST_MEMFD_FLAG_INIT_SHARED at the time of use and adding uffd_ prefixes to the callbacks make sense to me. I tested your changes in my local setup and they are functional with minor tweaks:
- remove the no-longer-used anon_can_userfault
- fix for the vm_flags check
mm/userfaultfd.c +1 -9
@@ -32,14 +32,6 @@ struct mfill_state {pmd_t *pmd; }; -static bool anon_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags) -{ -/* anonymous memory does not support MINOR mode */ -if (vm_flags &VM_UFFD_MINOR ) -return false; -return true; -} -static struct folio *anon_alloc_folio (struct vm_area_struct *vma,unsigned long addr) {
@@ -2051,7 +2043,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !ops->get_folio_noalloc)return false; -return ops->supported_uffd_flags & vm_flags; +return (ops->supported_uffd_flags & vm_flags) == vm_flags; } static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
Quoted 183 lines
include/linux/ renamed from include/linux/userfaultfd_k.h +0
-0
userfaultfd_k.hindex6f33307c2780..8a2d0625ffa3100644 --- a/include/linux/userfaultfd_k.h+++b/include/linux/userfaultfd_k.h@@ -82,8 +82,8 @@ externvm_fault_thandle_userfault(struct vm_fault *vmf, unsigned long reason);/* VMA userfaultfd operations */ struct
vm_uffd_ops{ - /*
Checks if a VMA can support userfaultfd */ - bool
(*can_userfault)(structvm_area_struct*vma,vm_flags_tvm_flags);
+ /* What UFFD flags/modes are supported. */ + constvm_flags_tsupported_uffd_flags; /* * Called to resolveUFFDIO_CONTINUErequest. * Should return the folio found at pgoff in the VMA's pagecache if itwith usage like:
static const struct
vm_uffd_opsshmem_uffd_ops= { .supported_uffd_flags =__VM_UFFD_FLAGS, .get_folio_noalloc =shmem_get_folio_noalloc, .alloc_folio = shmem_mfill_folio_alloc, .filemap_add = shmem_mfill_filemap_add, .filemap_remove =
shmem_mfill_filemap_remove, };[*] https://lore.kernel.org/all/[email protected]
Quoted 1 line
+ return true; +}
...
mm/filemap.c index
Quoted 6 lines
mm/filemap.c index renamed from mm/filemap.c +6
-0+static const struct vm_uffd_ops kvm_gmem_uffd_ops = { + .can_userfault = kvm_gmem_can_userfault, + .get_folio_noalloc = kvm_gmem_get_folio_noalloc, + .alloc_folio = kvm_gmem_folio_alloc, + .filemap_add = kvm_gmem_filemap_add, + .filemap_remove = kvm_gmem_filemap_remove,Please use
kvm_gmem_uffd_xxx(). The names are a bit verbose, but these are waaay to generic of names as-is, e.g.kvm_gmem_folio_alloc()has implications and restrictions far beyond just allocating a folio.All in all, somelike like so (completely untested):
---
include/linux/userfaultfd_k.h| 4 +-mm/filemap.c| 1 +mm/hugetlb.c| 8+--- mm/shmem.c | 7 +-- mm/ userfaultfd.c | 6 +-- virt/kvm/guest_memfd.c |80 ++++++++++++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 19 deletions(-)
include/linux/ renamed from include/linux/userfaultfd_k.h +0
-0
userfaultfd_k.hindex6f33307c2780..8a2d0625ffa3100644 --- a/include/linux/userfaultfd_k.h+++b/include/linux/userfaultfd_k.h@@ -82,8 +82,8 @@ externvm_fault_thandle_userfault(struct vm_fault *vmf, unsigned long reason);/* VMA userfaultfd operations */ struct
vm_uffd_ops{ - /*
Checks if a VMA can support userfaultfd */ - bool
(*can_userfault)(structvm_area_struct*vma,vm_flags_tvm_flags);
+ /* What UFFD flags/modes are supported. */ + constvm_flags_tsupported_uffd_flags; /* * Called to resolveUFFDIO_CONTINUErequest. * Should return the folio found at pgoff in
the VMA's pagecache if it diff --gita/mm/filemap.cb/mm/filemap.cindex6cd7974d4ada..19dfcebcd23f100644 ---a/mm/filemap.c+++ b/mm/filemap.c@@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio
*folio)
filemap_free_folio(mapping, folio); } +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");/* *
page_cache_delete_batch- delete several folios from page cache diff --gita/mm/hugetlb.cb/mm/hugetlb.cindex077968a8a69a..f55857961adb100644 ---a/mm/hugetlb.c+++ b/mm/hugetlb.c@@ -4819,14 +4819,8 @@ staticvm_fault_thugetlb_vm_op_fault(struct vm_fault *vmf)}#ifdef
CONFIG_USERFAULTFD-static bool hugetlb_can_userfault(structvm_area_struct*vma, -vm_flags_tvm_flags) -{ - return true; -} - static const structvm_uffd_opshugetlb_uffd_ops= { - .can_userfault =hugetlb_can_userfault, + .supported_uffd_flags =__VM_UFFD_FLAGS, }; #endifmm/shmem.c index renamed from mm/shmem.c +0
-0
239545352cd2..76d8488b9450100644 ---a/mm/shmem.c+++b/mm/shmem.cmm/shmem.c index renamed from mm/shmem.c +0
-1@@ -3250,13 +3250,8 @@ static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) return folio; } -static bool shmem_can_userfault(struct vm_area_struct *vma,
vm_flags_tvm_flags) -{ - return true; -} - static const
structvm_uffd_opsshmem_uffd_ops= {mm/shmem.c index renamed from mm/shmem.c +1
-1- .can_userfault = shmem_can_userfault, + .supported_uffd_flags = __VM_UFFD_FLAGS,.get_folio_noalloc =
shmem_get_folio_noalloc, .alloc_folio =shmem_mfill_folio_alloc, .filemap_add =shmem_mfill_filemap_add, diff --git a/mm/userfaultfd.cb/mm/userfaultfd.cindex9ba6ec8c0781..ccbd7bb334c2100644 ---a/mm/userfaultfd.c+++b/mm/userfaultfd.c@@ -58,8 +58,8 @@ static struct folio *anon_alloc_folio(structvm_area_struct*vma, }static const struct
vm_uffd_opsanon_uffd_ops= {mm/shmem.c index renamed from mm/shmem.c +2
-2- .can_userfault = anon_can_userfault, - .alloc_folio = anon_alloc_folio, + .supported_uffd_flags = __VM_UFFD_FLAGS & ~VM_UFFD_MINOR, + .alloc_folio = anon_alloc_folio, };static const struct
vm_uffd_ops*vma_uffd_ops(structvm_area_struct*vma) @@ -2055,7 +2055,7 @@ boolvma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, !ops->get_folio_noalloc)return false;- return
ops->can_userfault(vma, vm_flags); + return ops-Quoted 1 line
supported_uffd_flags&vm_flags; }static void userfaultfd_set_vm_flags(struct
vm_area_struct*vma, diff --gita/virt/kvm/guest_memfd.cb/virt/kvm/guest_memfd.cindex462c5c5cb602..e634bf671d12100644 ---a/virt/kvm/guest_memfd.c+++ b/virt/kvm/guest_memfd.c@@ -7,6 +7,7 @@ #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/userfaultfd_k.h>#include "
kvm_mm.h"mm/shmem.c index renamed from mm/shmem.c +0
-0@@ -59,6 +60,11 @@ static pgoff_t kvm_gmem_get_index(struct
kvm_memory_slot*slot,gfn_tgfn) return gfn -slot->base_gfn+ slot-Quoted 1 line
gmem.pgoff; }
+static bool kvm_gmem_is_shared_mem(struct inode *inode,
pgoff_t
index) +{ + return GMEM_I(inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED; +} + static int
__kvm_gmem_prepare_folio(struct kvm *kvm, structkvm_memory_slot
*slot,pgoff_tindex, struct folio *folio) { @@ -396,7 +402,7 @@
staticvm_fault_tkvm_gmem_fault_user_mapping(struct vm_fault *vmf)if (((loff_t)vmf->pgoff <<PAGE_SHIFT) >=i_size_read(inode)) returnVM_FAULT_SIGBUS;mm/shmem.c index renamed from mm/shmem.c +1
-1- if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)) + if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) return
VM_FAULT_SIGBUS;folio =
kvm_gmem_get_folio(inode, vmf->pgoff); @@ -456,12 +462,84 @@ static struct mempolicy *kvm_gmem_get_policy(structvm_area_struct*vma, } #endif /*CONFIG_NUMA*/+#ifdef
CONFIG_USERFAULTFD+static struct folio *kvm_gmem_uffd_get_folio_noalloc(struct inode *inode,mm/shmem.c index renamed from mm/shmem.c +3
-0+ pgoff_t pgoff) +{ + if (!kvm_gmem_is_shared_mem(inode, pgoff)) + return NULL; + + return
filemap_lock_folio(inode->i_mapping, pgoff); +} + +static struct folio *kvm_gmem_uffd_folio_alloc(structvm_area_struct*vma,mm/shmem.c index renamed from mm/shmem.c +2
-0+ unsigned long addr) +{ + struct inode *inode = file_inode(vma->vm_file); +
pgoff_tpgoff =linear_page_index(vma, addr); + struct
mempolicy *mpol; + struct folio *folio; +gfp_tgfp; +mm/shmem.c index renamed from mm/shmem.c +2
-0+ if (unlikely(pgoff >= (i_size_read(inode) >> PAGE_SHIFT))) + return NULL; + + if (!
kvm_gmem_is_shared_mem(inode, pgoff)) + return NULL; +
+ gfp =mapping_gfp_mask(inode->i_mapping); + mpol =mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff); +
mpol = mpol ?:get_task_policy(current); + folio =filemap_alloc_folio(gfp, 0, mpol); +mpol_cond_put(mpol); +
+ return folio; +} + +static int
kvm_gmem_uffd_filemap_add(struct folio *folio,mm/shmem.c index renamed from mm/shmem.c +2
-0+ struct vm_area_struct *vma, + unsigned long addr) +{ +struct inode *inode =
file_inode(vma->vm_file); + structaddress_space*mapping =inode->i_mapping; +pgoff_tpgoff =linear_page_index(vma, addr); + int err; + +__folio_set_locked(folio); + err = filemap_add_folio(mapping,
folio, pgoff,GFP_KERNEL); + if (err) { +folio_unlock(folio); + return err; + } + +
return 0; +} + +static void kvm_gmem_uffd_filemap_remove(struct
folio *folio, + structvm_area_struct*vma) +{ +filemap_remove_folio(folio); +folio_unlock(folio); +} + +static const structvm_uffd_opskvm_gmem_uffd_ops= { + .supported_uffd_flags =__VM_UFFD_FLAGS, + .get_folio_noalloc =kvm_gmem_uffd_get_folio_noalloc, + .alloc_folio =kvm_gmem_uffd_folio_alloc, + .filemap_add = kvm_gmem_uffd_filemap_add, + .filemap_remove =
kvm_gmem_uffd_filemap_remove, +}; +#endif /*CONFIG_USERFAULTFD*/ + static const structvm_operations_structkvm_gmem_vm_ops= { .fault =kvm_gmem_fault_user_mapping, #ifdefCONFIG_NUMA.get_policy =kvm_gmem_get_policy, .set_policy =kvm_gmem_set_policy, #endif +#ifdefCONFIG_USERFAULTFD
+ .uffd_ops =&kvm_gmem_uffd_ops, +#endif };static int
kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)base-commit:
d63beb006dba56d5fa219f106c7a97eb128c356f--
Mike Rapoport
maintainerRe: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
In reply to Sean ChristophersonHi Sean,
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Mike Rapoport
(quoted diff) +10
-0+#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) +{ + struct inode *inode = file_inode(vma->vm_file); + + /* + * Only support userfaultfd for guest_memfd with INIT_SHARED flag. + * This ensures the memory can be mapped to userspace. + */ + if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ return false;I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.
I'm not against solving it as a part of uffd support, but since we are very close to the merge window, for now I asked Andrew to drop drop guest_memfd patches from the set and only move forward with the refactoring of uffd and shmem that has value on its own.
Quoted 14 lines
Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.
My objection to an early, global "
can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. forVM_UFFD_MISSINGandVM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in__do_userfault(), not broadly whenVM_UFFD_MINORis added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path inkvm_gmem_fault_user_mapping().include/linux/userfaultfd_k.h +0
-0
include/linux/userfaultfd_k.h extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason); · line 86
Quoted 7 lines
include/linux/userfaultfd_k.h +2
-2@@ -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);/* VMA userfaultfd operations */ struct vm_uffd_ops { -/* Checks if a VMA can support userfaultfd */ -bool (*can_userfault)(struct vm_area_struct *vma,vm_flags_t vm_flags); +/* What UFFD flags/modes are supported. */ +const vm_flags_t supported_uffd_flags;
VMA maintainers really didn't like a fields flag in vm_uffd_ops when it was proposed earlier, but an indirect call may work.
Quoted 16 lines
include/linux/userfaultfd_k.h +0
-0/* * Called to resolve UFFDIO_CONTINUE request. * Should return the folio found at pgoff in the VMA's pagecache if it with usage like:
static const struct
vm_uffd_opsshmem_uffd_ops= {
.supported_uffd_flags =__VM_UFFD_FLAGS,
.get_folio_noalloc =shmem_get_folio_noalloc,
.alloc_folio =shmem_mfill_folio_alloc,
.filemap_add =shmem_mfill_filemap_add,
.filemap_remove =shmem_mfill_filemap_remove,
};All in all, somelike like so (completely untested):
Quoted 8 lines
--- include/linux/userfaultfd_k.h | 4 +- mm/filemap.c | 1 + mm/hugetlb.c | 8 +--- mm/shmem.c | 7 +-- mm/userfaultfd.c | 6 +-- virt/kvm/guest_memfd.c | 80 ++++++++++++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 19 deletions(-)
Let's revisit after -rc1 :)
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
In reply to Sean ChristophersonQuoted 12 lines · Sean Christopherson
[...snip...]
Review
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) +{ + struct inode *inode = file_inode(vma->vm_file); + + /* + * Only support userfaultfd for guest_memfd with INIT_SHARED flag. + * This ensures the memory can be mapped to userspace. + */
Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?
Quoted 10 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)) + return false;I'm not comfortable with this change. It works for now, but it's going to be wildly wrong when in-place conversion comes along. While I agree with the "Let's solve each problem in it's time :)"[*], the time for in-place conversion is now. In-place conversion isn't landing this cycle or next, but it's been in development for longer than UFFD support, and I'm not willing to punt solvable problems to that series, because it's plenty fat as is.
Thanks :)
Quoted 9 lines
Happily, IIUC, this is an easy problem to solve, and will have a nice side effect for the common UFFD code.
My objection to an early, global "
can_userfault()" check is that it's guaranteed to cause TOCTOU issues. E.g. forVM_UFFD_MISSINGandVM_UFFD_MINOR, the check on whether or not a given address can be faulted in needs to happen in__do_userfault(), not broadly whenVM_UFFD_MINORis added to a VMA. Conceptually, that also better aligns the code with the "normal" user fault path inkvm_gmem_fault_user_mapping().
Quoted 7 lines
I'm definitely not asking to fully prep for in-place conversion, I just want to set us up for success and also to not have to churn a pile of code. Concretely, again IIUC, I think we just need to move the
INIT_SHAREDcheck to ->alloc_folio() and ->get_folio_noalloc(). And if we extractkvm_gmem_is_shared_mem()now instead of waiting for in-place conversion, then we'll avoid a small amount of churn when in-place conversion comes along.
I think it would help if we do some refactoring of existing allocation functions to provide guest_memfd's uffd ->alloc_folio and ->get_folio_noalloc callbacks instead of having new functions.
Perhaps uffd ->alloc_folio and ->get_folio_noalloc can call an alloc function refactored out of kvm_gmem_get_folio(), and also do the kvm_gmem_is_shared_mem() check.
[email protected], who is starting work on guest_memfd preservation (as in LUO/KHO).
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
[...snip...]
Quoted 12 lines
Quoted 9 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) +{ + struct inode *inode = file_inode(vma->vm_file); + + /* + * Only support userfaultfd for guest_memfd with INIT_SHARED flag. + * This ensures the memory can be mapped to userspace. + */Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?
Yes? It's more the opposite though: memory that can't be SHARED and thus can't be mapped into userspace can't possibly support userfaultfd. Six of one, half dozen of the other, but I think it's important to highlight that there's no judgment call being made, i.e. we aren't deciding to support only SHARED just because we're lazy, it's a hard requirement.
Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations
Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
[...snip...]
Quoted 12 lines
Quoted 9 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_USERFAULTFD +static bool kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) +{ + struct inode *inode = file_inode(vma->vm_file); + + /* + * Only support userfaultfd for guest_memfd with INIT_SHARED flag. + * This ensures the memory can be mapped to userspace. + */Is the principle here that any memory that is allowed to be mapped to userspace can be userfault-ed?
Quoted 5 lines
Yes? It's more the opposite though: memory that can't be SHARED and thus can't be mapped into userspace can't possibly support userfaultfd. Six of one, half dozen of the other, but I think it's important to highlight that there's no judgment call being made, i.e. we aren't deciding to support only SHARED just because we're lazy, it's a hard requirement.
I guess this series is only intended to support some kinds of UFFD, but IIUC technically UFFDIO_CONTINUE can be used after the private memory is inserted (e.g. for TDX live migration).
Mike Rapoport
maintainerPATCH v4 14/15 KVM: selftests: test userfaultfd minor for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +113 -0
@@ -10,13 +10,17 @@#include <errno.h> #include <stdio.h> #include <fcntl.h> +#include <pthread.h> #include <linux/bitmap.h> #include <linux/falloc.h> #include <linux/sizes.h> +#include <linux/userfaultfd.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> #include "kvm_util.h" #include "numaif.h"
@@ -329,6 +333,112 @@ static void test_create_guest_memfd_multiple(struct kvm_vm *vm)close (fd1); } +struct fault_args { +char *addr; +char value; +}; + +static void *fault_thread_fn (void *arg) +{ +struct fault_args *args = arg; + +/* Trigger page fault */ + args->value = *args->addr; +return NULL ; +} + +static void test_uffd_minor (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_continue uffd_cont; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *mem_nofault, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xcd ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + +/* Map the guest_memfd twice: once with UFFD registered, once without */ + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + mem_nofault =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem_nofault !=MAP_FAILED ,"mmap should succeed" ); + +/* Register UFFD_MINOR on the first mapping */ + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MINOR ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + +/* +* Populate the page in the page cache first via mem_nofault. +* This is required for UFFD_MINOR - the page must exist in the cache. +* Write test data to the page. +*/ +memcpy (mem_nofault + offset, buf + offset, page_size); + +/* +* Now access the same page via mem (which has UFFD_MINOR registered). +* Since the page exists in the cache, this should trigger UFFD_MINOR. +*/ + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_MINOR , +"pagefault should be minor fault" ); + +/* Resolve the minor fault with UFFDIO_CONTINUE */ + uffd_cont.range.start = (unsigned long )fault_addr; + uffd_cont.range.len = page_size; + uffd_cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &uffd_cont); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_CONTINUE) should succeed" ); + +/* Wait for the faulting thread to complete */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* Verify the thread read the correct value */ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was written" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem_nofault, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -383,6 +493,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (file_size, vm, flags);gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); + +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +gmem_test (uffd_minor, vm, flags); }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.53.0
Mike Rapoport
maintainerPATCH v4 15/15 KVM: selftests: test userfaultfd missing for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a missing userfaultfd event in guest_memfd can be resolved via a UFFDIO_COPY ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/guest_memfd_test.c +79 -1
@@ -439,6 +439,82 @@ static void test_uffd_minor(int fd, size_t total_size)close (uffd); } +static void test_uffd_missing (int fd,size_t total_size) +{ +struct uffdio_register uffd_reg; +struct uffdio_copy uffd_copy; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; +const char test_val =0xab ; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); +memset (buf, test_val, total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =0 , + }; + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MISSING ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); +TEST_ASSERT (!(msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ), +"pagefault should not be write-protect" ); + + uffd_copy.dst = (unsigned long )fault_addr; + uffd_copy.src = (unsigned long )(buf + offset); + uffd_copy.len = page_size; + uffd_copy.mode =0 ; + ret =ioctl (uffd,UFFDIO_COPY , &uffd_copy); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_COPY) should succeed" ); + +/* Wait for the faulting thread to complete - this provides the memory barrier */ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +/* +* Now it's safe to check args.value - the thread has completed +* and memory is synchronized +*/ +TEST_ASSERT (args.value == test_val, +"memory should contain the value that was copied" ); +TEST_ASSERT (*(char *)(mem + offset) == test_val, +"no further fault is expected" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +static void test_guest_memfd_flags (struct kvm_vm *vm) {uint64_t valid_flags =vm_check_cap (vm,KVM_CAP_GUEST_MEMFD_FLAGS );
@@ -494,8 +570,10 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)gmem_test (fallocate, vm, flags);gmem_test (invalid_punch_hole, vm, flags); -if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) +if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) {gmem_test (uffd_minor, vm, flags); +gmem_test (uffd_missing, vm, flags); + } }static void test_guest_memfd (unsigned long vm_type)
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 1 line · On Thu, 2 Apr 2026 07:11:41 +0300 Mike Rapoport <[email protected]> wrote:
These patches enable support for userfaultfd in
guest_memfd.
Thanks, I updated mm.git to this version.
Quoted 2 lines
v4 changes:
* fix lock leak not only in patch 4, but also in patch 2, thanks Harry
And.... this resulted in no overall alteration to mm.git, which broke my very surprised script!
iirc, this series was to fix a little bisection hole, so this is to be expected?