mm, kvm: add guest_memfd support for uffd minor faults
Subsystems (8)
Mike Rapoport
maintainercover mm, kvm: add guest_memfd support for uffd minor faults
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches allow guest_memfd to notify userspace about minor page faults using userfaultfd and let userspace to resolve these page faults using UFFDIO_CONTINUE.
To allow UFFDIO_CONTINUE outside of the core mm I added a get_pagecache_folio() callback to vm_ops that allows an address space backing a VMA to return a folio that exists in it's page cache (patch 2)
In order for guest_memfd to notify userspace about page faults, it has to call handle_userfault() and since guest_memfd may be a part of kvm module, handle_userfault() is exported for kvm module (patch 3).
Note that patch 3 changelog does not provide motivation for enabling uffd in guest_memfd, mainly because I can't say I understand why is that required :)
Would be great to hear from KVM folks about it.
This series is the minimal change I've been able to come up with to allow integration of guest_memfd with uffd and while refactoring uffd and making mfill_atomic() flow more linear would have been a nice improvement, it's way out of the scope of enabling uffd with guest_memfd.
Mike Rapoport (Microsoft) (3):
userfaultfd: move vma_can_userfault out of line userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE userfaultfd, guest_memfd: support userfault minor mode in guest_memfd
Nikita Kalyazin (1):
KVM: selftests: test userfaultfd minor for guest_memfd
fs/userfaultfd.c | 4 +- include/linux/mm.h | 9 ++ include/linux/userfaultfd_k.h | 36 +----- include/uapi/linux/userfaultfd.h | 8 +- mm/shmem.c | 20 ++++ mm/userfaultfd.c | 88 ++++++++++++--- .../testing/selftests/kvm/guest_memfd_test.c | 103 ++++++++++++++++++ virt/kvm/guest_memfd.c | 30 +++++ 8 files changed, 245 insertions(+), 53 deletions(-)
base-commit: 6146a0f1dfae5d37442a9ddcba012add260bceb0
Signature · 1 line
2.50.1
Mike Rapoport
maintainerRFC PATCH 1/4 userfaultfd: move vma_can_userfault out of line
From: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault() has grown pretty big and it's not called on performance critical path.
Move it out of line.
No functional changes.
--- include/linux/userfaultfd_k.h | 36 ++--------------------------------- mm/userfaultfd.c | 34 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 34 deletions(-)
include/linux/userfaultfd_k.h +2 -34
@@ -208,40 +208,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -vm_flags_t vm_flags, -bool wp_async) -{ - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true; - -#ifndef CONFIG_PTE_MARKER_UFFD_WP -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) -return false; -#endif - -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async);static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {
mm/userfaultfd.c +34 -0
@@ -1977,6 +1977,40 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,return moved ? moved : err; } +bool vma_can_userfault (struct vm_area_struct *vma,vm_flags_t vm_flags, +bool wp_async) +{ + vm_flags &= __VM_UFFD_FLAGS; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + +if ((vm_flags &VM_UFFD_MINOR ) && + (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) +return false; + +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +#ifndef CONFIG_PTE_MARKER_UFFD_WP +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then shmem & hugetlbfs are not supported but only +* anonymous. +*/ +if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) +return false; +#endif + +/* By default, allow any of anon|shmem|hugetlb */ +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +static void userfaultfd_set_vm_flags (struct vm_area_struct *vma,vm_flags_t vm_flags) {
Signature · 1 line
2.50.1
Re: [RFC PATCH 1/4] userfaultfd: move vma_can_userfault out of line
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
vma_can_userfault()has grown pretty big and it's not called on performance critical path.Move it out of line.
No functional changes.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
Signature · 3 lines
Cheers David
Mike Rapoport
maintainerRFC PATCH 2/4 userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE it needs to get a folio that already exists in the pagecache backing that VMA.
Instead of using shmem_get_folio() for that, add a get_pagecache_folio() method to 'struct vm_operations_struct' that will return a folio if it exists in the VMA's pagecache at given pgoff.
Implement get_pagecache_folio() method for shmem and slightly refactor userfaultfd's mfill_atomic() and mfill_atomic_pte_continue() to support this new API.
--- include/linux/mm.h | 9 +++++++ mm/shmem.c | 20 ++++++++++++++++ mm/userfaultfd.c | 60 ++++++++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 20 deletions(-)
include/linux/mm.h +9 -0
@@ -690,6 +690,15 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +/* +* Called by userfault to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +*/ +struct folio *(*get_pagecache_folio)(struct vm_area_struct *vma, +pgoff_t pgoff); +#endif };#ifdef CONFIG_NUMA_BALANCING
mm/shmem.c +20 -0
@@ -3260,6 +3260,20 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,shmem_inode_unacct_blocks (inode,1 ); return ret; } + +static struct folio *shmem_get_pagecache_folio (struct vm_area_struct *vma, +pgoff_t pgoff) +{ +struct inode *inode =file_inode (vma->vm_file); +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +}#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS
@@ -5292,6 +5306,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .get_pagecache_folio = shmem_get_pagecache_folio, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5301,6 +5318,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .get_pagecache_folio = shmem_get_pagecache_folio, +#endif };int shmem_init_fs_context (struct fs_context *fc)
mm/userfaultfd.c +40 -20
99 lines not on this page · raw message
Signature · 1 line
2.50.1
David Hildenbrand (Red Hat)
maintainerRe: [RFC PATCH 2/4] userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE
Quoted 26 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with
UFFDIO_CONTINUEit needs to get a folio that already exists in the pagecache backing that VMA.Instead of using
shmem_get_folio()for that, add aget_pagecache_folio()method to 'structvm_operations_struct' that will return a folio if it exists in the VMA's pagecache at given pgoff.Implement
get_pagecache_folio()method for shmem and slightly refactor userfaultfd'smfill_atomic()andmfill_atomic_pte_continue()to support this new API.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/mm.h | 9 +++++++ mm/shmem.c | 20 ++++++++++++++++ mm/userfaultfd.c | 60 ++++++++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 20 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h struct vm_operations_struct { · line 698
Quoted 9 lines
include/linux/mm.h +6
-0@@ -690,6 +690,15 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD +/* +* Called by userfault to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +*/
What are the locking +refcount rules? Without looking at the code, I would assume we return with a folio reference held and the folio locked?
include/linux/mm.h struct vm_operations_struct { · line 700
Quoted 2 lines
include/linux/mm.h +2
-0+struct folio *(*get_pagecache_folio)(struct vm_area_struct *vma, +pgoff_t pgoff);
The combination of VMA + pgoff looks weird at first. Would vma + addr or vma+vma_offset into vma be better?
But it also makes me wonder if the callback would ever even require the VMA, or actually only vma->vm_file?
Thinking out loud, I wonder if one could just call that "get_folio" or "get_shared_folio" (IOW, never an anon folio in a MAP_PRIVATE mapping).
Quoted 99 lines
include/linux/mm.h +1
-0+#endif };#ifdef CONFIG_NUMA_BALANCING mm/shmem.c +20
-0@@ -3260,6 +3260,20 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd,shmem_inode_unacct_blocks (inode,1 ); return ret; } + +static struct folio *shmem_get_pagecache_folio (struct vm_area_struct *vma, +pgoff_t pgoff) +{ +struct inode *inode =file_inode (vma->vm_file); +struct folio *folio; +int err; + + err =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (err) +return ERR_PTR (err); + +return folio; +}#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS @@ -5292,6 +5306,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .get_pagecache_folio = shmem_get_pagecache_folio, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {@@ -5301,6 +5318,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .get_pagecache_folio = shmem_get_pagecache_folio, +#endif };int shmem_init_fs_context (struct fs_context *fc)mm/userfaultfd.c +8
-13@@ -382,21 +382,17 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, unsigned long dst_addr, uffd_flags_t flags) { -struct inode *inode =file_inode (dst_vma->vm_file);pgoff_t pgoff =linear_page_index (dst_vma, dst_addr);struct folio *folio;struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); + folio = dst_vma->vm_ops->get_pagecache_folio (dst_vma, pgoff);/* Our caller expects us to return -EFAULT if we failed to find folio */ -if (ret == -ENOENT ) - ret = -EFAULT ; -if (ret) - goto out; -if (!folio) { - ret = -EFAULT ; - goto out; +if (IS_ERR_OR_NULL (folio)) { +if (PTR_ERR (folio) == -ENOENT || !folio) +return -EFAULT ; +return PTR_ERR (folio); } page =folio_file_page (folio, pgoff);@@ -411,13 +407,12 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, goto out_release;folio_unlock (folio); - ret =0 ; -out: -return ret; +return 0 ; + out_release:folio_unlock (folio);folio_put (folio); - goto out; +return ret; }/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ @@ -694,6 +689,22 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd,return err;
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, · line 699
Quoted 10 lines
mm/userfaultfd.c +8
-0} +static __always_inline bool vma_can_mfill_atomic(struct vm_area_struct *vma, +uffd_flags_t flags) +{ +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { +if (vma->vm_ops && vma->vm_ops->get_pagecache_folio) +return true; +else +return false;
Probably easier to read is
return vma->vm_ops && vma->vm_ops->get_pagecache_folio;
mm/userfaultfd.c static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, · line 705
Quoted 6 lines
mm/userfaultfd.c +6
-0+ } + +if (vma_is_anonymous (vma) ||vma_is_shmem (vma)) +return true; + +return false;
Could also be simplified to:
return vma_is_anonymous(vma) || vma_is_shmem(vma);
Signature · 3 lines
Cheers David
Mike Rapoport
maintainerRe: [RFC PATCH 2/4] userfaultfd, shmem: use a VMA callback to handle UFFDIO_CONTINUE
Quoted 31 lines · David Hildenbrand (Red Hat)
Quoted 30 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When userspace resolves a page fault in a shmem VMA with
UFFDIO_CONTINUEit needs to get a folio that already exists in the pagecache backing that VMA.Instead of using
shmem_get_folio()for that, add aget_pagecache_folio()method to 'structvm_operations_struct' that will return a folio if it exists in the VMA's pagecache at given pgoff.Implement
get_pagecache_folio()method for shmem and slightly refactor userfaultfd'smfill_atomic()andmfill_atomic_pte_continue()to support this new API.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- include/linux/mm.h | 9 +++++++ mm/shmem.c | 20 ++++++++++++++++ mm/userfaultfd.c | 60 ++++++++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 20 deletions(-)include/linux/mm.h +1
-0@@ -690,6 +690,15 @@ struct vm_operations_struct {struct page *(*find_normal_page)(struct vm_area_struct *vma,unsigned long addr);#endif /* CONFIG_FIND_NORMAL_PAGE */ +#ifdef CONFIG_USERFAULTFD
Quoted 8 lines
include/linux/mm.h struct vm_operations_struct { · line 698
Quoted 5 lines
include/linux/mm.h +5
-0+/* +* Called by userfault to resolve UFFDIO_CONTINUE request. +* Should return the folio found at pgoff in the VMA's pagecache if it +* exists or ERR_PTR otherwise. +*/ What are the locking +refcount rules? Without looking at the code, I would assume we return with a folio reference held and the folio locked?
Right, will add it to the comment
Quoted 6 lines
include/linux/mm.h struct vm_operations_struct { · line 700
Quoted 2 lines
include/linux/mm.h +2
-0+struct folio *(*get_pagecache_folio)(struct vm_area_struct *vma, +pgoff_t pgoff);The combination of VMA + pgoff looks weird at first. Would vma + addr or vma+vma_offset into vma be better?
Copied from map_pages() :)
Quoted 2 lines
But it also makes me wonder if the callback would ever even require the VMA, or actually only
vma->vm_file?
It's actually inode, I'm going to pass that instead of vma.
Quoted 2 lines
Thinking out loud, I wonder if one could just call that "
get_folio" or "get_shared_folio" (IOW, never an anon folio in aMAP_PRIVATEmapping).
Naming is hard :)
get_shared_folio() sounds good to me so unless there other suggestions I'll stick with it.
Quoted 3 lines
Quoted 3 lines
include/linux/mm.h +1
-0+#endif };#ifdef CONFIG_NUMA_BALANCING
...
Quoted 23 lines
include/linux/mm.h
Quoted 8 lines
include/linux/mm.h +8
-0+static __always_inline bool vma_can_mfill_atomic(struct vm_area_struct *vma, +uffd_flags_t flags) +{ +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) { +if (vma->vm_ops && vma->vm_ops->get_pagecache_folio) +return true; +else +return false;Probably easier to read is
return vma->vm_ops && vma->vm_ops->get_pagecache_folio;include/linux/mm.h
Quoted 6 lines
include/linux/mm.h +6
-0+ } + +if (vma_is_anonymous (vma) ||vma_is_shmem (vma)) +return true; + +return false;Could also be simplified to:
Quoted 1 line
return
vma_is_anonymous(vma)||vma_is_shmem(vma);
Agree with for both of them.
Quoted 4 lines
Signature · 3 lines
Cheers David
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerRFC PATCH 3/4 userfaultfd, guest_memfd: support userfault minor mode in guest_memfd
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
* Export handle_userfault() for KVM module so that fault() handler in guest_memfd would be able to notify userspace about page faults in its address space.
* Implement get_pagecache_folio() for guest_memfd.
* And finally, introduce UFFD_FEATURE_MINOR_GENERIC that will allow using userfaultfd minor mode with memory types other than shmem and hugetlb provided they are allowed to call handle_userfault() and implement get_pagecache_folio().
--- fs/userfaultfd.c | 4 +++- include/uapi/linux/userfaultfd.h | 8 +++++++- virt/kvm/guest_memfd.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 2 deletions(-)
fs/userfaultfd.c +3 -1
@@ -537,6 +537,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) out:return ret; } +EXPORT_SYMBOL_FOR_MODULES (handle_userfault,"kvm" );static void userfaultfd_event_wait_completion (struct userfaultfd_ctx *ctx,struct userfaultfd_wait_queue *ewq)
@@ -1978,7 +1979,8 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, uffdio_api.features =UFFD_API_FEATURES ;#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= - ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM ); + ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM | +UFFD_FEATURE_MINOR_GENERIC );#endif #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP ;
include/uapi/linux/userfaultfd.h +7 -1
@@ -42,7 +42,8 @@ UFFD_FEATURE_WP_UNPOPULATED | \ UFFD_FEATURE_POISON | \ UFFD_FEATURE_WP_ASYNC | \ - UFFD_FEATURE_MOVE) + UFFD_FEATURE_MOVE | \ + UFFD_FEATURE_MINOR_GENERIC)#define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -210,6 +211,10 @@ struct uffdio_api {* UFFD_FEATURE_MINOR_SHMEM indicates the same support as * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. * +* UFFD_FEATURE_MINOR_GENERIC indicates that minor faults can be +* intercepted for file-backed memory in case subsystem backing this +* memory supports it. +* * UFFD_FEATURE_EXACT_ADDRESS indicates that the exact address of page * faults would be provided and the offset within the page would not be * masked.
@@ -248,6 +253,7 @@ struct uffdio_api {#define UFFD_FEATURE_POISON (1<<14)#define UFFD_FEATURE_WP_ASYNC (1<<15)#define UFFD_FEATURE_MOVE (1<<16) +#define UFFD_FEATURE_MINOR_GENERIC (1<<17)__u64 features;__u64 ioctls;
virt/kvm/guest_memfd.c +30 -0
@@ -4,6 +4,7 @@#include <linux/kvm_host.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h"
@@ -369,6 +370,12 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)return vmf_error (err); } +if (userfaultfd_minor (vmf->vma)) { +folio_unlock (folio); +folio_put (folio); +return handle_userfault (vmf,VM_UFFD_MINOR ); + } +if (WARN_ON_ONCE (folio_test_large (folio))) { ret =VM_FAULT_SIGBUS ; goto out_folio;
@@ -390,8 +397,31 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)return ret; } +#ifdef CONFIG_USERFAULTFD +static struct folio *kvm_gmem_get_pagecache_folio (struct vm_area_struct *vma, +pgoff_t pgoff) +{ +struct inode *inode =file_inode (vma->vm_file); +struct folio *folio; + + folio =kvm_gmem_get_folio (inode, pgoff); +if (IS_ERR_OR_NULL (folio)) +return folio; + +if (!folio_test_uptodate (folio)) { +clear_highpage (folio_page (folio,0 )); +kvm_gmem_mark_prepared (folio); + } + +return folio; +} +#endif +static const struct vm_operations_struct kvm_gmem_vm_ops = { .fault = kvm_gmem_fault_user_mapping, +#ifdef CONFIG_USERFAULTFD + .get_pagecache_folio = kvm_gmem_get_pagecache_folio, +#endif };static int kvm_gmem_mmap (struct file *file,struct vm_area_struct *vma)
Signature · 1 line
2.50.1
David Hildenbrand (Red Hat)
maintainerRe: [RFC PATCH 3/4] userfaultfd, guest_memfd: support userfault minor mode in guest_memfd
Quoted 86 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
* Export
handle_userfault()for KVM module so thatfault()handler inguest_memfdwould be able to notify userspace about page faults in its address space.
* Implementget_pagecache_folio()forguest_memfd.
* And finally, introduceUFFD_FEATURE_MINOR_GENERICthat will allow using userfaultfd minor mode with memory types other than shmem and hugetlb provided they are allowed to callhandle_userfault()and implementget_pagecache_folio().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- fs/userfaultfd.c | 4 +++- include/uapi/linux/userfaultfd.h | 8 +++++++- virt/kvm/guest_memfd.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 2 deletions(-)fs/userfaultfd.c +3
-1@@ -537,6 +537,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) out:return ret; } +EXPORT_SYMBOL_FOR_MODULES (handle_userfault,"kvm" );static void userfaultfd_event_wait_completion (struct userfaultfd_ctx *ctx,struct userfaultfd_wait_queue *ewq)@@ -1978,7 +1979,8 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, uffdio_api.features =UFFD_API_FEATURES ;#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= - ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM ); + ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM | +UFFD_FEATURE_MINOR_GENERIC );#endif #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP ;include/uapi/linux/userfaultfd.h +7
-1@@ -42,7 +42,8 @@ UFFD_FEATURE_WP_UNPOPULATED | \ UFFD_FEATURE_POISON | \ UFFD_FEATURE_WP_ASYNC | \ - UFFD_FEATURE_MOVE) + UFFD_FEATURE_MOVE | \ + UFFD_FEATURE_MINOR_GENERIC)#define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \@@ -210,6 +211,10 @@ struct uffdio_api {* UFFD_FEATURE_MINOR_SHMEM indicates the same support as * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. * +* UFFD_FEATURE_MINOR_GENERIC indicates that minor faults can be +* intercepted for file-backed memory in case subsystem backing this +* memory supports it. +* * UFFD_FEATURE_EXACT_ADDRESS indicates that the exact address of page * faults would be provided and the offset within the page would not be * masked. @@ -248,6 +253,7 @@ struct uffdio_api {#define UFFD_FEATURE_POISON (1<<14)#define UFFD_FEATURE_WP_ASYNC (1<<15)#define UFFD_FEATURE_MOVE (1<<16) +#define UFFD_FEATURE_MINOR_GENERIC (1<<17)__u64 features;__u64 ioctls;virt/kvm/guest_memfd.c +1
-0@@ -4,6 +4,7 @@#include <linux/kvm_host.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h"
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) · line 377
Quoted 8 lines
virt/kvm/guest_memfd.c +5
-0@@ -369,6 +370,12 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)return vmf_error (err); } +if (userfaultfd_minor (vmf->vma)) { +folio_unlock (folio); +folio_put (folio); +return handle_userfault (vmf,VM_UFFD_MINOR ); + }
Staring at things like VM_FAULT_NEEDDSYNC, I'm wondering whether we could have a new return value from ->fault that would indicate that handle_userfault(vmf, VM_UFFD_MINOR) should be called.
Maybe some VM_FAULT_UFFD_MINOR or simply VM_FAULT_USERFAULTFD and we can just derive that it is VM_UFFD_MINOR.
include/linux/mm_types.h +4 -0
@@ -1601,6 +1601,8 @@ typedef __bitwise unsigned int vm_fault_t;* fsync() to complete (for synchronous page faults * in DAX) * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released +* @VM_FAULT_USERFAULTFD: ->fault did not modify page tables and needs +* handle_userfault() to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -1618,6 +1620,7 @@ enum vm_fault_reason {VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 ,VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , +VM_FAULT_USERFAULTFD = (__force vm_fault_t )0x006000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
@@ -1642,6 +1645,7 @@ enum vm_fault_reason { { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ + { VM_FAULT_USERFAULTFD, "USERFAULTFD" },\ { VM_FAULT_COMPLETED, "COMPLETED" } struct vm_special_mapping {
IIUC, we have exactly two invocations of ->fault(vmf) in memory.c where we would have to handle it IIUC. And the return value would never leave the core.
That way, we wouldn't have to export handle_userfault().
Just a thought ...
Signature · 3 lines
Cheers David
Mike Rapoport
maintainerRe: [RFC PATCH 3/4] userfaultfd, guest_memfd: support userfault minor mode in guest_memfd
Quoted 24 lines · David Hildenbrand (Red Hat)
Quoted 11 lines · Mike Rapoport
virt/kvm/guest_memfd.c +1
-0@@ -4,6 +4,7 @@#include <linux/kvm_host.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> +#include <linux/userfaultfd_k.h> #include "kvm_mm.h" virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) · line 376
Quoted 7 lines
virt/kvm/guest_memfd.c +5
-0@@ -369,6 +370,12 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)return vmf_error (err); } +if (userfaultfd_minor (vmf->vma)) { +folio_unlock (folio); +folio_put (folio); +return handle_userfault (vmf,VM_UFFD_MINOR ); + }Staring at things like
VM_FAULT_NEEDDSYNC, I'm wondering whether we could have a new return value from ->fault that would indicate thathandle_userfault(vmf, VM_UFFD_MINOR)should be called.
Quoted 2 lines
Maybe some
VM_FAULT_UFFD_MINORor simplyVM_FAULT_USERFAULTFDand we can just derive that it isVM_UFFD_MINOR.
_UFFD_MINOR sounds better, maybe we'll want something for missing later on.
Quoted 29 lines
include/linux/mm_types.h +4
-0@@ -1601,6 +1601,8 @@ typedef __bitwise unsigned int vm_fault_t;* fsync() to complete (for synchronous page faults * in DAX) * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released +* @VM_FAULT_USERFAULTFD: ->fault did not modify page tables and needs +* handle_userfault() to complete * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */ @@ -1618,6 +1620,7 @@ enum vm_fault_reason {VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 ,VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , +VM_FAULT_USERFAULTFD = (__force vm_fault_t )0x006000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };@@ -1642,6 +1645,7 @@ enum vm_fault_reason { { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ + { VM_FAULT_USERFAULTFD, "USERFAULTFD" },\ { VM_FAULT_COMPLETED, "COMPLETED" } struct vm_special_mapping {
Quoted 3 lines
IIUC, we have exactly two invocations of ->fault(vmf) in
memory.cwhere we would have to handle it IIUC. And the return value would never leave the core.
I've found only one :/
But nevertheless, I like the idea to return VM_FAULT_UFFD_MINOR from ->fault() and then call handle_userfault() from __do_fault().
Quoted 8 lines
That way, we wouldn't have to export
handle_userfault().Just a thought ...
Signature · 3 lines
Cheers David
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerRFC PATCH 4/4 KVM: selftests: test userfaultfd minor for guest_memfd
In reply to Mike RapoportFrom: Nikita Kalyazin <[email protected]>
The test demonstrates that a minor userfaultfd event in guest_memfd can be resolved via a memcpy followed by a UFFDIO_CONTINUE ioctl.
--- .../testing/selftests/kvm/guest_memfd_test.c | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +103 -0
@@ -10,13 +10,17 @@#include <errno.h> #include <stdio.h> #include <fcntl.h> +#include <pthread.h> #include <linux/bitmap.h> #include <linux/falloc.h> #include <linux/sizes.h> +#include <linux/userfaultfd.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> #include "kvm_util.h" #include "test_util.h"
@@ -254,6 +258,104 @@ static void test_guest_memfd_flags(struct kvm_vm *vm) } } +struct fault_args { +char *addr; +volatile char value; +}; + +static void *fault_thread_fn (void *arg) +{ +struct fault_args *args = arg; + +/* Trigger page fault */ + args->value = *args->addr; +return NULL ; +} + +static void test_uffd_minor (int fd,size_t total_size) +{ +struct uffdio_api uffdio_api = { + .api =UFFD_API , + .features =UFFD_FEATURE_MINOR_GENERIC , + }; +struct uffdio_register uffd_reg; +struct uffdio_continue uffd_cont; +struct uffd_msg msg; +struct fault_args args; +pthread_t fault_thread; +void *mem, *mem_nofault, *buf =NULL ; +int uffd, ret; +off_t offset = page_size; +void *fault_addr; + + ret =posix_memalign (&buf, page_size, total_size); +TEST_ASSERT_EQ (ret,0 ); + +memset (buf,0xaa , total_size); + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC ); +TEST_ASSERT (uffd !=-1 ,"userfaultfd creation should succeed" ); + + ret =ioctl (uffd,UFFDIO_API , &uffdio_api); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_API) should succeed" ); + + mem =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem !=MAP_FAILED ,"mmap should succeed" ); + + mem_nofault =mmap (NULL , total_size,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +TEST_ASSERT (mem_nofault !=MAP_FAILED ,"mmap should succeed" ); + + uffd_reg.range.start = (unsigned long )mem; + uffd_reg.range.len = total_size; + uffd_reg.mode =UFFDIO_REGISTER_MODE_MINOR ; + ret =ioctl (uffd,UFFDIO_REGISTER , &uffd_reg); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_REGISTER) should succeed" ); + + ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , + offset, page_size); +TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) should succeed" ); + + fault_addr = mem + offset; + args.addr = fault_addr; + + ret =pthread_create (&fault_thread,NULL , fault_thread_fn, &args); +TEST_ASSERT (ret ==0 ,"pthread_create should succeed" ); + + ret =read (uffd, &msg,sizeof (msg)); +TEST_ASSERT (ret !=-1 ,"read from userfaultfd should succeed" ); +TEST_ASSERT (msg.event ==UFFD_EVENT_PAGEFAULT ,"event type should be pagefault" ); +TEST_ASSERT ((void *)(msg.arg.pagefault.address & ~(page_size -1 )) == fault_addr, +"pagefault should occur at expected address" ); + +memcpy (mem_nofault + offset, buf + offset, page_size); + + uffd_cont.range.start = (unsigned long )fault_addr; + uffd_cont.range.len = page_size; + uffd_cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &uffd_cont); +TEST_ASSERT (ret !=-1 ,"ioctl(UFFDIO_CONTINUE) should succeed" ); + +/* +* wait for fault_thread to finish to make sure fault happened and was +* resolved before we verify the values +*/ + ret =pthread_join (fault_thread,NULL ); +TEST_ASSERT (ret ==0 ,"pthread_join should succeed" ); + +TEST_ASSERT (args.value == *(char *)(mem_nofault + offset), +"memory should contain the value that was copied" ); +TEST_ASSERT (args.value == *(char *)(mem + offset), +"no further fault is expected" ); + + ret =munmap (mem_nofault, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); + + ret =munmap (mem, total_size); +TEST_ASSERT (!ret,"munmap should succeed" ); +free (buf); +close (uffd); +} +#define gmem_test (__test, __vm, __flags) \ do { \ int fd = vm_create_guest_memfd(__vm, page_size * 4, __flags); \
@@ -273,6 +375,7 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags)if (flags &GUEST_MEMFD_FLAG_INIT_SHARED ) {gmem_test (mmap_supported, vm, flags);gmem_test (fault_overflow, vm, flags); +gmem_test (uffd_minor, vm, flags); }else {gmem_test (fault_private, vm, flags); }
Signature · 1 line
2.50.1
Quoted 12 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
These patches allow
guest_memfdto notify userspace about minor page faults using userfaultfd and let userspace to resolve these page faults usingUFFDIO_CONTINUE.To allow
UFFDIO_CONTINUEoutside of the core mm I added aget_pagecache_folio()callback tovm_opsthat allows an address space backing a VMA to return a folio that exists in it's page cache (patch 2)
Quoted 8 lines
In order for
guest_memfdto notify userspace about page faults, it has to callhandle_userfault()and sinceguest_memfdmay be a part of kvm module,handle_userfault()is exported for kvm module (patch 3).Note that patch 3 changelog does not provide motivation for enabling uffd in
guest_memfd, mainly because I can't say I understand why is that required :)
Would be great to hear from KVM folks about it.
Hi Mike,
Thanks for posting it!
In our use case, Firecracker snapshot-restore using UFFD [1], we will use UFFD minor/continue to respond to guest_memfd faults in user mappings primarily due to VMM accesses that are required for PV (virtio) device emulation and also KVM accesses when decoding MMIO operations on x86.
Nikita
Quoted 29 lines
This series is the minimal change I've been able to come up with to allow integration of
guest_memfdwith uffd and while refactoring uffd and makingmfill_atomic()flow more linear would have been a nice improvement, it's way out of the scope of enabling uffd withguest_memfd.Mike Rapoport (Microsoft) (3):
userfaultfd: movevma_can_userfaultout of line userfaultfd, shmem: use a VMA callback to handleUFFDIO_CONTINUEuserfaultfd,guest_memfd: support userfault minor mode inguest_memfdNikita Kalyazin (1):
KVM: selftests: test userfaultfd minor forguest_memfdfs/userfaultfd.c | 4 +- include/linux/mm.h | 9 ++ include/linux/userfaultfd_k.h | 36 +----- include/uapi/linux/userfaultfd.h | 8 +- mm/shmem.c | 20 ++++ mm/userfaultfd.c | 88 ++++++++++++--- .../testing/selftests/kvm/guest_memfd_test.c | 103 ++++++++++++++++++ virt/kvm/guest_memfd.c | 30 +++++ 8 files changed, 245 insertions(+), 53 deletions(-)base-commit:
6146a0f1dfae5d37442a9ddcba012add260bceb0--
2.50.1
Peter Xu
reviewerQuoted 4 lines · Nikita Kalyazin
In our use case, Firecracker snapshot-restore using UFFD [1], we will use UFFD minor/continue to respond to
guest_memfdfaults in user mappings primarily due to VMM accesses that are required for PV (virtio) device emulation and also KVM accesses when decoding MMIO operations on x86.
I'm curious if firecracker plans to support live snapshot save. When with something like ioctls_supported flags, guest-memfd can declare support for wr-protect support easily too, and synchronous userfaultfd wr-protect traps will be an efficient way to do live save.
I'm guessing it's not an immediate demand now or it would have been asked already supporting both MINOR and WP, but I just want to raise this question. Qemu already supports live snapshot save, so it'll always be good gmem can also support wp at some point, but it can be done later too.
Thanks,
Signature · 1 line
Peter Xu
Quoted 11 lines · Peter Xu
Quoted 4 lines · Nikita Kalyazin
In our use case, Firecracker snapshot-restore using UFFD [1], we will use UFFD minor/continue to respond to
guest_memfdfaults in user mappings primarily due to VMM accesses that are required for PV (virtio) device emulation and also KVM accesses when decoding MMIO operations on x86.I'm curious if firecracker plans to support live snapshot save. When with something like
ioctls_supportedflags, guest-memfd can declare support for wr-protect support easily too, and synchronous userfaultfd wr-protect traps will be an efficient way to do live save.
Quoted 4 lines
I'm guessing it's not an immediate demand now or it would have been asked already supporting both MINOR and WP, but I just want to raise this question. Qemu already supports live snapshot save, so it'll always be good gmem can also support wp at some point, but it can be done later too.
No, live snapshots haven't been on our plan so far.
Quoted 6 lines
Thanks,
--
Peter Xu