mm/userfaultfd: modulize memory types
Subsystems (7)
Peter Xu
reviewercover mm/userfaultfd: modulize memory types
[based on akpm/mm-new]
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]
This is not yet relevant to any guest-memfd support, but paving way for it. Here, the major goal is to make kernel modules be able to opt-in with any form of userfaultfd supports, like guest-memfd. This alternative option should hopefully be cleaner, and avoid leaking userfault details into vm_ops.fault().
It also means this series does not depend on anything. It's a pure refactoring of userfaultfd internals to provide a generic API, so that other types of files, especially RAM based, can support userfaultfd without touching mm/ at all.
To achieve it, this series introduced a file operation called vm_uffd_ops. The ops needs to be provided when a file type supports any of userfaultfd.
With that, I moved both hugetlbfs and shmem over.
Hugetlbfs is still very special that it will only use partial of the vm_uffd_ops API, due to similar reason why hugetlb_vm_op_fault() has a BUG() and so far hard-coded into core mm. But this should still be better, because at least hugetlbfs is still always involved in feature probing (e.g. where it used to not support ZEROPAGE and we have a hard-coded line to fail that, and some more). Meanwhile after this series, shmem will be completely converted to the new vm_uffd_ops API; the final vm_uffd_ops for shmem looks like this:
static const vm_uffd_ops shmem_uffd_ops = {
.uffd_features = __VM_UFFD_FLAGS,
.uffd_ioctls = BIT(_UFFDIO_COPY) |
BIT(_UFFDIO_ZEROPAGE) |
BIT(_UFFDIO_WRITEPROTECT) |
BIT(_UFFDIO_CONTINUE) |
BIT(_UFFDIO_POISON),
.uffd_get_folio = shmem_uffd_get_folio,
.uffd_copy = shmem_mfill_atomic_pte,
};As I mentioned in one of my reply to Nikita, I don't like the current interface of uffd_copy(), but this will be the minimum change version of such API to support complete extrenal-module-ready userfaultfd. Here, very minimal change will be needed from shmem side to support that.
Meanwhile, the vm_uffd_ops is also not the only place one will need to provide to support userfaultfd. Normally vm_ops.fault() will also need to be updated, but that's a generic function and it'll play together with the new vm_uffd_ops to make everything fly.
No functional change expected at all after the whole series applied. There might be some slightly stricter check on uffd ops here and there in the last patch, but that really shouldn't stand out anywhere to anyone.
For testing: besides the cross-compilation tests, I did also try with uffd-stress in a VM to measure any perf difference before/after the change; The static call becomes a pointer now. I really cannot measure anything different, which is more or less expected.
Comments welcomed, thanks.
Peter Xu (4):
mm: Introduce vm_uffd_ops API
mm/shmem: Support vm_uffd_ops API
mm/hugetlb: Support vm_uffd_ops API
mm: Apply vm_uffd_ops API to core mm
include/linux/mm.h | 71 +++++++++++++++++++++ include/linux/shmem_fs.h | 14 ----- include/linux/userfaultfd_k.h | 58 ++++------------- mm/hugetlb.c | 19 ++++++ mm/shmem.c | 28 ++++++++- mm/userfaultfd.c | 115 +++++++++++++++++++++++++--------- 6 files changed, 217 insertions(+), 88 deletions(-)
Signature · 1 line
2.49.0
Peter Xu
reviewerPATCH 1/4 mm: Introduce vm_uffd_ops API
Introduce a generic userfaultfd API for vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifying the core files. More importantly, when the module can be compiled out of the kernel.
So, instead of having core mm referencing modules that may not ever exist, we need to have modules opt-in on core mm hooks instead.
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define vm_uffd_ops, instead of changing anything in core mm.
Note that such API will not work for anonymous. Core mm will process anonymous memory separately for userfault operations like before.
This patch only introduces the API alone so that we can start to move existing users over but without breaking them.
Currently the uffd_copy() API is almost designed to be the simplistic with minimum mm changes to move over to the API.
--- include/linux/mm.h | 71 +++++++++++++++++++++++++++++++++++ include/linux/userfaultfd_k.h | 12 ------ 2 files changed, 71 insertions(+), 12 deletions(-)
include/linux/mm.h +71 -0
@@ -576,6 +576,70 @@ struct vm_fault {*/ }; +#ifdef CONFIG_USERFAULTFD +/* A combined operation mode + behavior flags. */ +typedef unsigned int __bitwise uffd_flags_t ; + +enum mfill_atomic_mode { +MFILL_ATOMIC_COPY , +MFILL_ATOMIC_ZEROPAGE , +MFILL_ATOMIC_CONTINUE , +MFILL_ATOMIC_POISON , +NR_MFILL_ATOMIC_MODES , +}; + +/* VMA userfaultfd operations */ +typedef struct { +/** +* @uffd_features: features supported in bitmask. +* +* When the ops is defined, the driver must set non-zero features +* to be a subset (or all) of: VM_UFFD_MISSING|WP|MINOR. +*/ +unsigned long uffd_features; +/** +* @uffd_ioctls: ioctls supported in bitmask. +* +* Userfaultfd ioctls supported by the module. Below will always +* be supported by default whenever a module provides vm_uffd_ops: +* +* _UFFDIO_API, _UFFDIO_REGISTER, _UFFDIO_UNREGISTER, _UFFDIO_WAKE +* +* The module needs to provide all the rest optionally supported +* ioctls. For example, when VM_UFFD_MISSING was supported, +* _UFFDIO_COPY must be supported as ioctl, while _UFFDIO_ZEROPAGE +* is optional. +*/ +unsigned long uffd_ioctls; +/** +* uffd_get_folio: Handler to resolve UFFDIO_CONTINUE request. +* +* @inode: the inode for folio lookup +* @pgoff: the pgoff of the folio +* @folio: returned folio pointer +* +* Return: zero if succeeded, negative for errors. +*/ +int (*uffd_get_folio)(struct inode *inode,pgoff_t pgoff, +struct folio **folio); +/** +* uffd_copy: Handler to resolve UFFDIO_COPY|ZEROPAGE request. +* +* @dst_pmd: target pmd to resolve page fault +* @dst_vma: target vma +* @dst_addr: target virtual address +* @src_addr: source address to copy from +* @flags: userfaultfd request flags +* @foliop: previously allocated folio +* +* Return: zero if succeeded, negative for errors. +*/ +int (*uffd_copy)(pmd_t *dst_pmd,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +uffd_flags_t flags,struct folio **foliop); +}vm_uffd_ops ; +#endif +/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer
@@ -653,6 +717,13 @@ struct vm_operations_struct {*/ struct page *(*find_special_page)(struct vm_area_struct *vma,unsigned long addr); +#ifdef CONFIG_USERFAULTFD +/* +* Userfaultfd related ops. Modules need to define this to support +* userfaultfd. +*/ +const vm_uffd_ops *userfaultfd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING
include/linux/userfaultfd_k.h +0 -12
@@ -80,18 +80,6 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); -/* A combined operation mode + behavior flags. */ -typedef unsigned int __bitwise uffd_flags_t ; - -/* Mutually exclusive modes of operation. */ -enum mfill_atomic_mode { -MFILL_ATOMIC_COPY , -MFILL_ATOMIC_ZEROPAGE , -MFILL_ATOMIC_CONTINUE , -MFILL_ATOMIC_POISON , -NR_MFILL_ATOMIC_MODES , -}; -#define MFILL_ATOMIC_MODE_BITS (const_ilog2(NR_MFILL_ATOMIC_MODES - 1) + 1)#define MFILL_ATOMIC_BIT (nr) BIT(MFILL_ATOMIC_MODE_BITS + (nr))#define MFILL_ATOMIC_FLAG (nr) ((__force uffd_flags_t) MFILL_ATOMIC_BIT(nr))
Signature · 1 line
2.49.0
Mike Rapoport
maintainerRe: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Hi Peter,
Quoted 88 lines · Peter Xu
Introduce a generic userfaultfd API for
vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifying the core files. More importantly, when the module can be compiled out of the kernel.So, instead of having core mm referencing modules that may not ever exist, we need to have modules opt-in on core mm hooks instead.
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm.Note that such API will not work for anonymous. Core mm will process anonymous memory separately for userfault operations like before.
This patch only introduces the API alone so that we can start to move existing users over but without breaking them.
Currently the
uffd_copy()API is almost designed to be the simplistic with minimum mm changes to move over to the API.Signed-off-by Peter Xu <[email protected]>--- include/linux/mm.h | 71 +++++++++++++++++++++++++++++++++++ include/linux/userfaultfd_k.h | 12 ------ 2 files changed, 71 insertions(+), 12 deletions(-)include/linux/mm.h +53
-0@@ -576,6 +576,70 @@ struct vm_fault {*/ }; +#ifdef CONFIG_USERFAULTFD +/* A combined operation mode + behavior flags. */ +typedef unsigned int __bitwise uffd_flags_t; + +enum mfill_atomic_mode { + MFILL_ATOMIC_COPY, + MFILL_ATOMIC_ZEROPAGE, + MFILL_ATOMIC_CONTINUE, + MFILL_ATOMIC_POISON, + NR_MFILL_ATOMIC_MODES, +}; + +/* VMA userfaultfd operations */ +typedef struct { + /** + * @uffd_features: features supported in bitmask. + * + * When the ops is defined, the driver must set non-zero features + * to be a subset (or all) of: VM_UFFD_MISSING|WP|MINOR. + */ + unsigned long uffd_features; + /** + * @uffd_ioctls: ioctls supported in bitmask. + * + * Userfaultfd ioctls supported by the module. Below will always + * be supported by default whenever a module provides vm_uffd_ops: + * + * _UFFDIO_API, _UFFDIO_REGISTER, _UFFDIO_UNREGISTER, _UFFDIO_WAKE + * + * The module needs to provide all the rest optionally supported + * ioctls. For example, when VM_UFFD_MISSING was supported, + * _UFFDIO_COPY must be supported as ioctl, while _UFFDIO_ZEROPAGE + * is optional. + */ + unsigned long uffd_ioctls; + /** + * uffd_get_folio: Handler to resolve UFFDIO_CONTINUE request. + * + * @inode: the inode for folio lookup + * @pgoff: the pgoff of the folio + * @folio: returned folio pointer + * + * Return: zero if succeeded, negative for errors. + */ + int (*uffd_get_folio)(struct inode *inode, pgoff_t pgoff, + struct folio **folio); + /** + * uffd_copy: Handler to resolve UFFDIO_COPY|ZEROPAGE request. + * + * @dst_pmd: target pmd to resolve page fault + * @dst_vma: target vma + * @dst_addr: target virtual address + * @src_addr: source address to copy from
include/linux/mm.h struct vm_fault { · line 641
Quoted 10 lines
include/linux/mm.h +10
-0+* @flags: userfaultfd request flags +* @foliop: previously allocated folio +* +* Return: zero if succeeded, negative for errors. +*/ +int (*uffd_copy)(pmd_t *dst_pmd,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, + uffd_flags_t flags, struct folio **foliop); +} vm_uffd_ops; +#endif
Can't we define vm_uffd_ops in userfaultfd_k.h?
A forward declaration in mm.h should suffice and modules that want to use uffd can include userfaultfd_k.h.
Quoted 44 lines
include/linux/mm.h +8
-0+/* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer @@ -653,6 +717,13 @@ struct vm_operations_struct {*/ struct page *(*find_special_page)(struct vm_area_struct *vma,unsigned long addr); +#ifdef CONFIG_USERFAULTFD +/* +* Userfaultfd related ops. Modules need to define this to support +* userfaultfd. +*/ +const vm_uffd_ops *userfaultfd_ops; +#endif };#ifdef CONFIG_NUMA_BALANCING include/linux/userfaultfd_k.h +0
-12@@ -80,18 +80,6 @@ struct userfaultfd_ctx {extern vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason); -/* A combined operation mode + behavior flags. */ -typedef unsigned int __bitwise uffd_flags_t ; - -/* Mutually exclusive modes of operation. */ -enum mfill_atomic_mode { -MFILL_ATOMIC_COPY , -MFILL_ATOMIC_ZEROPAGE , -MFILL_ATOMIC_CONTINUE , -MFILL_ATOMIC_POISON , -NR_MFILL_ATOMIC_MODES , -}; -#define MFILL_ATOMIC_MODE_BITS (const_ilog2(NR_MFILL_ATOMIC_MODES - 1) + 1)#define MFILL_ATOMIC_BIT (nr) BIT(MFILL_ATOMIC_MODE_BITS + (nr))#define MFILL_ATOMIC_FLAG (nr) ((__force uffd_flags_t) MFILL_ATOMIC_BIT(nr))Signature · 1 line
2.49.0
Signature · 2 lines
Sincerely yours, Mike.
Peter Xu
reviewerRe: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 7 lines · Mike Rapoport
include/linux/mm.h struct vm_fault { · line 641
Quoted 2 lines
include/linux/mm.h +2
-0+} vm_uffd_ops; +#endif Can't we define
vm_uffd_opsinuserfaultfd_k.h?A forward declaration in
mm.hshould suffice and modules that want to use uffd can includeuserfaultfd_k.h.
Good point, I'll do that, thanks!
Signature · 1 line
Peter Xu
Re: [PATCH 1/4] mm: Introduce vm_uffd_ops API
In reply to Peter XuOn 20.06.25 21:03, Peter Xu wrote:
Hi Peter,
Quoted 2 lines
Introduce a generic userfaultfd API for
vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifying
The sentence is confusing ("vma ... as a module").
Did you mean something like ".. so that a vma that is backed by a special-purpose in-memory filesystem like shmem or hugetlb can support userfaultfd without modifying the uffd core; this is required when the in-memory filesystem is built as a module."
Quoted 9 lines
the core files. More importantly, when the module can be compiled out of the kernel.
So, instead of having core mm referencing modules that may not ever exist, we need to have modules opt-in on core mm hooks instead.
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm.
Talking about modules that much is a bit confusing. I think this is more about cleanly supporting in-memory filesystems, without the need to special-case each and every one of them; can be viewed a cleanup independent of the module requirement from guest_memfd.
Quoted 10 lines
Note that such API will not work for anonymous. Core mm will process anonymous memory separately for userfault operations like before.
This patch only introduces the API alone so that we can start to move existing users over but without breaking them.
Currently the
uffd_copy()API is almost designed to be the simplistic with minimum mm changes to move over to the API.
Is there a way to move part of the actual implementation (how this is all wired up) from patch #4 into this patch, to then only remove the old shmem/hugetlb hooks (that are effectively unused) in patch #4?
Signature · 3 lines
Cheers, David / dhildenb
Peter Xu
reviewerRe: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 3 lines · David Hildenbrand
On 20.06.25 21:03, Peter Xu wrote:
Hi Peter,
Hey David,
Quoted 10 lines
Quoted 3 lines
Introduce a generic userfaultfd API for
vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifyingThe sentence is confusing ("vma ... as a module").
Did you mean something like ".. so that a vma that is backed by a special-purpose in-memory filesystem like shmem or hugetlb can support userfaultfd without modifying the uffd core; this is required when the in-memory filesystem is built as a module."
I wanted to avoid mentioning of "in-memory file systems" here.
How about an updated commit like this?
Currently, most of the userfaultfd features are implemented directly in the core mm. It will invoke VMA specific functions whenever necessary. So far it is fine because it almost only interacts with shmem and hugetlbfs.
This patch introduces a generic userfaultfd API for vm_operations_struct, so that any type of file (including kernel modules that can be compiled separately from the kernel core) can support userfaults without modifying the core files.
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define vm_uffd_ops, instead of changing anything in core mm.
...
Quoted 7 lines
Quoted 7 lines
the core files. More importantly, when the module can be compiled out of the kernel.
So, instead of having core mm referencing modules that may not ever exist, we need to have modules opt-in on core mm hooks instead.
Quoted 8 lines
Quoted 3 lines
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm.Talking about modules that much is a bit confusing. I think this is more about cleanly supporting in-memory filesystems, without the need to special-case each and every one of them; can be viewed a cleanup independent of the module requirement from
guest_memfd.
Yes. But if we don't need to support kernel modules actually we don't need this.. IMHO it's so far really about cleanly support kernel modules, which can even be out-of-tree (though that's not my purpose of the change..).
Please help check if above updated commit message would be better.
Quoted 8 lines
Quoted 8 lines
Note that such API will not work for anonymous. Core mm will process anonymous memory separately for userfault operations like before.
This patch only introduces the API alone so that we can start to move existing users over but without breaking them.
Quoted 7 lines
Quoted 3 lines
Currently the
uffd_copy()API is almost designed to be the simplistic with minimum mm changes to move over to the API.Is there a way to move part of the actual implementation (how this is all wired up) from patch #4 into this patch, to then only remove the old shmem/hugetlb hooks (that are effectively unused) in patch #4?
Not much I really removed on the hooks, but I was trying to reuse almost existing functions. Here hugetlb is almost untouched on hooks, then I reused the shmem existing function for uffd_copy() rather than removing it (I did need to remove the definition in the shmem header though becuse it's not needed to be exported).
The major thing got removed in patch 4 was some random checks over uffd ops and vma flags. I intentionally made them all in patch 4 to make review possible. Otherwise it can be slightly awkward to reason what got removed without knowing what is protecting those checks.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 11 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
On 20.06.25 21:03, Peter Xu wrote:
Hi Peter,
Hey David,
Quoted 4 lines
Quoted 3 lines
Introduce a generic userfaultfd API for
vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifying
Quoted 8 lines
Quoted 6 lines
The sentence is confusing ("vma ... as a module").
Did you mean something like ".. so that a vma that is backed by a special-purpose in-memory filesystem like shmem or hugetlb can support userfaultfd without modifying the uffd core; this is required when the in-memory filesystem is built as a module."
I wanted to avoid mentioning of "in-memory file systems" here.
I thought one of the challenges of supporting guest_memfd on anything that is not a special in-memory file system is also related to how the pagecache handles readahead.
So ...
Quoted 11 lines
How about an updated commit like this?
Currently, most of the userfaultfd features are implemented directly in the core mm. It will invoke VMA specific functions whenever necessary. So far it is fine because it almost only interacts with shmem and hugetlbfs.
This patch introduces a generic userfaultfd API for
vm_operations_struct, so that any type of file (including kernel modules that can be compiled separately from the kernel core) can support userfaults without modifying the core files.
.... is it really "any file" ? I doubt it, but you likely have a better idea on how it all could just work with "any file".
Quoted 6 lines
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm....
Talking about files and modules is still confusing I'm afraid. It's really a special-purpose file (really, not any ordinary files on ordinary filesystems), no?
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
the core files. More importantly, when the module can be compiled out of the kernel.
So, instead of having core mm referencing modules that may not ever exist, we need to have modules opt-in on core mm hooks instead.
Quoted 8 lines
Quoted 3 lines
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm.Talking about modules that much is a bit confusing. I think this is more about cleanly supporting in-memory filesystems, without the need to special-case each and every one of them; can be viewed a cleanup independent of the module requirement from
guest_memfd.
Quoted 5 lines
Yes. But if we don't need to support kernel modules actually we don't need this.. IMHO it's so far really about cleanly support kernel modules, which can even be out-of-tree (though that's not my purpose of the change..).
Please help check if above updated commit message would be better.
I agree that another special-purpose file (like implemented by guest_memfd) would need that. But if we could get rid of "hugetlb"/"shmem" special-casing in userfaultfd, it would be a rasonable independent cleanup.
But I can spot in patch #3 now:
"Hugetlbfs still has its own hard-coded handler in userfaultfd, due to limitations similar to vm_operations_struct.fault(). TODO: generalize it to use the API function."
I would have hoped that we clean that up in one go instead.
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Note that such API will not work for anonymous. Core mm will process anonymous memory separately for userfault operations like before.
This patch only introduces the API alone so that we can start to move existing users over but without breaking them.
Quoted 7 lines
Quoted 3 lines
Currently the
uffd_copy()API is almost designed to be the simplistic with minimum mm changes to move over to the API.Is there a way to move part of the actual implementation (how this is all wired up) from patch #4 into this patch, to then only remove the old shmem/hugetlb hooks (that are effectively unused) in patch #4?
Not much I really removed on the hooks, but I was trying to reuse almost existing functions. Here hugetlb is almost untouched on hooks, then I reused the shmem existing function for
uffd_copy()rather than removing it (I did need to remove the definition in the shmem header though becuse it's not needed to be exported).
Quoted 4 lines
The major thing got removed in patch 4 was some random checks over uffd ops and vma flags. I intentionally made them all in patch 4 to make review possible. Otherwise it can be slightly awkward to reason what got removed without knowing what is protecting those checks.
Agreed. It's a shame the new API is not a proper replacement for hugetlb special casing just yet ...
Signature · 3 lines
Cheers, David / dhildenb
Peter Xu
reviewerRe: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 19 lines · David Hildenbrand
Quoted 18 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
On 20.06.25 21:03, Peter Xu wrote:
Hi Peter,
Hey David,
Quoted 10 lines
Quoted 3 lines
Introduce a generic userfaultfd API for
vm_operations_struct, so that one vma, especially when as a module, can support userfaults without modifyingThe sentence is confusing ("vma ... as a module").
Did you mean something like ".. so that a vma that is backed by a special-purpose in-memory filesystem like shmem or hugetlb can support userfaultfd without modifying the uffd core; this is required when the in-memory filesystem is built as a module."
Quoted 7 lines
Quoted 1 line
I wanted to avoid mentioning of "in-memory file systems" here.
I thought one of the challenges of supporting
guest_memfdon anything that is not a special in-memory file system is also related to how the pagecache handles readahead.So ...
See uffd_disable_fault_around(). We should make sure no such happens into pgtables when some special type of file is suppoorted, if it ever happens, besides shmem. IIUC readahead on page caches are fine for non-MISSING traps. So a file can support MINOR, for example, but then it'll also need to make sure all those aspected are well considered.
Quoted 21 lines
Quoted 4 lines
How about an updated commit like this?
Quoted 8 lines
Currently, most of the userfaultfd features are implemented directly in the core mm. It will invoke VMA specific functions whenever necessary. So far it is fine because it almost only interacts with shmem and hugetlbfs.
This patch introduces a generic userfaultfd API for
vm_operations_struct, so that any type of file (including kernel modules that can be compiled separately from the kernel core) can support userfaults without modifying the core files..... is it really "any file" ? I doubt it, but you likely have a better idea on how it all could just work with "any file".
Quoted 5 lines
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm.
Quoted 5 lines
Quoted 1 line
...Talking about files and modules is still confusing I'm afraid. It's really a special-purpose file (really, not any ordinary files on ordinary filesystems), no?
One major reason I wanted to avoid the term "in-memory" is that we already support most of the files on WP_ASYNC, so emphasizing on in-memory might be misleading, even though WP_ASYNC isn't much taken into the picture of the vm_uffd_ops being proposed.
The other thing is, besides the original form of userfaultfd (which is the MISSING traps), almost all the rest (sync-wp, continue, poison, maybe even MOVE but that's still more special) should be at least logically doable on most of the files like WP_ASYNC. When proposing this API, I wanted to make it as generic as possible when people reading about it. Hope that makes sense.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
How about an updated commit like this?
Currently, most of the userfaultfd features are implemented directly in the core mm. It will invoke VMA specific functions whenever necessary. So far it is fine because it almost only interacts with shmem and hugetlbfs.
This patch introduces a generic userfaultfd API for
vm_operations_struct, so that any type of file (including kernel modules that can be compiled separately from the kernel core) can support userfaults without modifying the core files..... is it really "any file" ? I doubt it, but you likely have a better idea on how it all could just work with "any file".
Quoted 6 lines
After this API applied, if a module wants to support userfaultfd, the module should only need to touch its own file and properly define
vm_uffd_ops, instead of changing anything in core mm....
Quoted 8 lines
Quoted 3 lines
Talking about files and modules is still confusing I'm afraid. It's really a special-purpose file (really, not any ordinary files on ordinary filesystems), no?
One major reason I wanted to avoid the term "in-memory" is that we already support most of the files on
WP_ASYNC, so emphasizing on in-memory might be misleading, even thoughWP_ASYNCisn't much taken into the picture of thevm_uffd_opsbeing proposed.
Oh, yes, agreed on WP_ASYNC. But they would not be using the vma_ops thingy, right?
Signature · 3 lines
Cheers, David / dhildenb
Peter Xu
reviewerRe: [PATCH 1/4] mm: Introduce vm_uffd_ops API
Quoted 2 lines · David Hildenbrand
Oh, yes, agreed on
WP_ASYNC. But they would not be using thevma_opsthingy, right?
Yes, currently WP_ASYNC bypassed that. So if something declares WP in the uffd_features in the new API it'll be only for sync (and it really should almost start working for most files too like what async did..).
Signature · 1 line
Peter Xu
Peter Xu
reviewerPATCH 2/4 mm/shmem: Support vm_uffd_ops API
In reply to Peter XuAdd support for the new vm_uffd_ops API for shmem. Note that this only introduces the support, the API is not yet used by core mm.
Due to the tailored uffd_copy() API, shmem is extremely easy to support it by reusing the existing mfill function.
It only needs a separate uffd_get_folio() definition but that's oneliner.
--- mm/shmem.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
mm/shmem.c +26 -0
@@ -3151,6 +3151,13 @@ static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap,#endif /* CONFIG_TMPFS_QUOTA */ #ifdef CONFIG_USERFAULTFD + +static int shmem_uffd_get_folio (struct inode *inode,pgoff_t pgoff, +struct folio **folio) +{ +return shmem_get_folio (inode, pgoff,0 , folio,SGP_NOALLOC ); +} + int shmem_mfill_atomic_pte(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -5194,6 +5201,19 @@ static int shmem_error_remove_folio(struct address_space *mapping,return 0 ; } +#ifdef CONFIG_USERFAULTFD +static const vm_uffd_ops shmem_uffd_ops = { + .uffd_features = __VM_UFFD_FLAGS, + .uffd_ioctls =BIT (_UFFDIO_COPY) | +BIT (_UFFDIO_ZEROPAGE) | +BIT (_UFFDIO_WRITEPROTECT) | +BIT (_UFFDIO_CONTINUE) | +BIT (_UFFDIO_POISON), + .uffd_get_folio = shmem_uffd_get_folio, + .uffd_copy = shmem_mfill_atomic_pte, +}; +#endif +static const struct address_space_operations shmem_aops = { .dirty_folio = noop_dirty_folio, #ifdef CONFIG_TMPFS
@@ -5296,6 +5316,9 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .userfaultfd_ops = &shmem_uffd_ops, +#endif };static const struct vm_operations_struct shmem_anon_vm_ops = {
@@ -5305,6 +5328,9 @@ static const struct vm_operations_struct shmem_anon_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy,#endif +#ifdef CONFIG_USERFAULTFD + .userfaultfd_ops = &shmem_uffd_ops, +#endif };int shmem_init_fs_context (struct fs_context *fc)
Signature · 1 line
2.49.0
Peter Xu
reviewerPATCH 3/4 mm/hugetlb: Support vm_uffd_ops API
In reply to Peter XuAdd support for the new vm_uffd_ops API for hugetlb. Note that this only introduces the support, the API is not yet used by core mm.
Due to legacy reasons, it's still not trivial to move hugetlb completely to the API (like shmem). But it will still use uffd_features and uffd_ioctls properly on the API because that's pretty general.
--- mm/hugetlb.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
mm/hugetlb.c +19 -0
@@ -5459,6 +5459,22 @@ static vm_fault_t hugetlb_vm_op_fault(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_USERFAULTFD +static const vm_uffd_ops hugetlb_uffd_ops = { + .uffd_features = __VM_UFFD_FLAGS, +/* _UFFDIO_ZEROPAGE not supported */ + .uffd_ioctls =BIT (_UFFDIO_COPY) | +BIT (_UFFDIO_WRITEPROTECT) | +BIT (_UFFDIO_CONTINUE) | +BIT (_UFFDIO_POISON), +/* +* Hugetlbfs still has its own hard-coded handler in userfaultfd, +* due to limitations similar to vm_operations_struct.fault(). +* TODO: generalize it to use the API functions. +*/ +}; +#endif +/* * When a new function is introduced to vm_operations_struct and added * to hugetlb_vm_ops, please consider adding the function to shm_vm_ops.
@@ -5472,6 +5488,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .close = hugetlb_vm_op_close, .may_split = hugetlb_vm_op_split, .pagesize = hugetlb_vm_op_pagesize, +#ifdef CONFIG_USERFAULTFD + .userfaultfd_ops = &hugetlb_uffd_ops, +#endif }; static pte_t make_huge_pte(struct vm_area_struct *vma, struct folio *folio,
Signature · 1 line
2.49.0
Peter Xu
reviewerPATCH 4/4 mm: Apply vm_uffd_ops API to core mm
In reply to Peter XuThis patch completely moves the old userfaultfd core to use the new vm_uffd_ops API. After this change, existing file systems will start to use the new API for userfault operations.
When at it, moving vma_can_userfault() into mm/userfaultfd.c instead, because it's getting too big. It's only used in slow paths so it shouldn't be an issue.
This will also remove quite some hard-coded checks for either shmem or hugetlbfs. Now all the old checks should still work but with vm_uffd_ops.
Note that anonymous memory will still need to be processed separately because it doesn't have vm_ops at all.
--- include/linux/shmem_fs.h | 14 ----- include/linux/userfaultfd_k.h | 46 ++++---------- mm/shmem.c | 2 +- mm/userfaultfd.c | 115 +++++++++++++++++++++++++--------- 4 files changed, 101 insertions(+), 76 deletions(-)
include/linux/shmem_fs.h +0 -14
@@ -195,20 +195,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages);extern void shmem_uncharge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that
include/linux/userfaultfd_k.h +13 -33
63 lines not on this page · raw message
mm/shmem.c +1 -1
@@ -3158,7 +3158,7 @@ static int shmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff,return shmem_get_folio (inode, pgoff,0 , folio,SGP_NOALLOC ); } -int shmem_mfill_atomic_pte(pmd_t *dst_pmd, +static int shmem_mfill_atomic_pte(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr,
mm/userfaultfd.c +87 -28
165 lines not on this page · raw message
Signature · 1 line
2.49.0
kernel test robot
build botRe: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm
kernel test robot noticed the following build warnings:
Hi Peter,
kernel test robot noticed the following build warnings:
[auto build test WARNING on next-20250620]
[cannot apply to akpm-mm/mm-everything linus/master v6.16-rc2 v6.16-rc1 v6.15 v6.16-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-Introduce-vm_uffd_ops-API/20250621-030557 base: next-20250620 patch link: https://lore.kernel.org/r/20250620190342.1780170-5-peterx%40redhat.com patch subject: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm config: i386-randconfig-061-20250622 (https://download.01.org/0day-ci/archive/20250623/[email protected]/config) compiler: gcc-11 (Debian 11.3.0-12) 11.3.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250623/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
mm/shmem.c: note: in included file (through include/linux/shmem_fs.h):
>> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t
--
mm/hugetlb.c:669:12: sparse: sparse: context imbalance in 'allocate_file_region_entries' - wrong count at exit mm/hugetlb.c:740:13: sparse: sparse: context imbalance in 'region_add' - wrong count at exit mm/hugetlb.c:807:13: sparse: sparse: context imbalance in 'region_chg' - wrong count at exit mm/hugetlb.c:5798:20: sparse: sparse: context imbalance in 'move_huge_pte' - different lock contexts for basic block mm/hugetlb.c: note: in included file:
include/linux/mm.h:1391:22: sparse: sparse: context imbalance in 'hugetlb_wp' - unexpected unlock mm/hugetlb.c: note: in included file (through include/linux/hugetlb.h, include/linux/migrate.h):
>> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t
--
mm/userfaultfd.c:270:9: sparse: sparse: context imbalance in 'mfill_atomic_install_pte' - different lock contexts for basic block mm/userfaultfd.c:412:9: sparse: sparse: context imbalance in 'mfill_atomic_pte_zeropage' - different lock contexts for basic block mm/userfaultfd.c:498:9: sparse: sparse: context imbalance in 'mfill_atomic_pte_poison' - different lock contexts for basic block mm/userfaultfd.c: note: in included file:
>> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t >> include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricted uffd_flags_t
mm/userfaultfd.c: note: in included file (through include/linux/rbtree.h, include/linux/mm_types.h, include/linux/mmzone.h, ...):
include/linux/rcupdate.h:871:25: sparse: sparse: context imbalance in 'move_pages_pte' - unexpected unlock
vim +90 include/linux/userfaultfd_k.h
87
88 static inline enum mfill_atomic_mode uffd_flags_get_mode(uffd_flags_t flags)
89 {
> 90 return (enum mfill_atomic_mode)(flags & MFILL_ATOMIC_MODE_MASK);
91 }
92 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Peter Xu
reviewerRe: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm
Quoted 64 lines · kernel test robot
Hi Peter,
kernel test robot noticed the following build warnings:
[auto build test WARNING on next-20250620]
[cannot apply to akpm-mm/mm-everything linus/master v6.16-rc2 v6.16-rc1 v6.15 v6.16-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-Introduce-vm_uffd_ops-API/20250621-030557 base: next-20250620 patch link: https://lore.kernel.org/r/20250620190342.1780170-5-peterx%40redhat.com patch subject: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm config: i386-randconfig-061-20250622 (https://download.01.org/0day-ci/archive/20250623/[email protected]/config) compiler: gcc-11 (Debian 11.3.0-12) 11.3.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250623/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/sparse warnings: (new ones prefixed by >>)
mm/shmem.c: note: in included file (throughinclude/linux/shmem_fs.h):Quoted 1 line
Quoted 1 line
include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_t--
mm/hugetlb.c:669:12: sparse: sparse: context imbalance in 'allocate_file_region_entries' - wrong count at exitmm/hugetlb.c:740:13: sparse: sparse: context imbalance in 'region_add' - wrong count at exitmm/hugetlb.c:807:13: sparse: sparse: context imbalance in 'region_chg' - wrong count at exitmm/hugetlb.c:5798:20: sparse: sparse: context imbalance in 'move_huge_pte' - different lock contexts for basic blockmm/hugetlb.c: note: in included file:
include/linux/mm.h:1391:22: sparse: sparse: context imbalance in 'hugetlb_wp' - unexpected unlockmm/hugetlb.c: note: in included file (throughinclude/linux/hugetlb.h,include/linux/migrate.h):Quoted 2 lines
Quoted 2 lines
include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_t--
mm/userfaultfd.c:270:9: sparse: sparse: context imbalance in 'mfill_atomic_install_pte' - different lock contexts for basic blockmm/userfaultfd.c:412:9: sparse: sparse: context imbalance in 'mfill_atomic_pte_zeropage' - different lock contexts for basic blockmm/userfaultfd.c:498:9: sparse: sparse: context imbalance in 'mfill_atomic_pte_poison' - different lock contexts for basic blockmm/userfaultfd.c: note: in included file:Quoted 12 lines
Quoted 12 lines
include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tQuoted 8 lines
Quoted 8 lines
include/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_tinclude/linux/userfaultfd_k.h:90:17: sparse: sparse: cast from restricteduffd_flags_t
mm/userfaultfd.c: note: in included file (throughinclude/linux/rbtree.h,include/linux/mm_types.h,include/linux/mmzone.h, ...):
include/linux/rcupdate.h:871:25: sparse: sparse: context imbalance in 'move_pages_pte' - unexpected unlock
Quoted 8 lines
vim +90
include/linux/userfaultfd_k.h87 88 static inline enum mfill_atomic_mode uffd_flags_get_mode(uffd_flags_t flags) 89 {Quoted 1 line
90 return (enum mfill_atomic_mode)(flags &
MFILL_ATOMIC_MODE_MASK);91 } 92
Sigh.. I thought the cast would helped to tell sparse on this. Looks like I need __force..
I'll squash below change when repost:
include/linux/userfaultfd_k.h +1 -1
@@ -87,7 +87,7 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);static inline enum mfill_atomic_mode uffd_flags_get_mode (uffd_flags_t flags) { -return (enum mfill_atomic_mode )(flags &MFILL_ATOMIC_MODE_MASK ); +return (__force enum mfill_atomic_mode )(flags &MFILL_ATOMIC_MODE_MASK ); }static inline bool uffd_flags_mode_is (uffd_flags_t flags,enum mfill_atomic_mode expected)
Signature · 1 line
Peter Xu
Re: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm
In reply to Peter XuQuoted 92 lines · Peter Xu
This patch completely moves the old userfaultfd core to use the new
vm_uffd_opsAPI. After this change, existing file systems will start to use the new API for userfault operations.When at it, moving
vma_can_userfault()intomm/userfaultfd.cinstead, because it's getting too big. It's only used in slow paths so it shouldn't be an issue.This will also remove quite some hard-coded checks for either shmem or hugetlbfs. Now all the old checks should still work but with
vm_uffd_ops.Note that anonymous memory will still need to be processed separately because it doesn't have
vm_opsat all.Signed-off-by Peter Xu <[email protected]>--- include/linux/shmem_fs.h | 14 ----- include/linux/userfaultfd_k.h | 46 ++++---------- mm/shmem.c | 2 +- mm/userfaultfd.c | 115 +++++++++++++++++++++++++--------- 4 files changed, 101 insertions(+), 76 deletions(-)include/linux/shmem_fs.h +0
-14@@ -195,20 +195,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)extern bool shmem_charge (struct inode *inode,long pages);extern void shmem_uncharge (struct inode *inode,long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte (pmd_t *dst_pmd, -struct vm_area_struct *dst_vma, -unsigned long dst_addr, -unsigned long src_addr, -uffd_flags_t flags, -struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte (dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ -/* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that include/linux/userfaultfd_k.h +7
-19@@ -85,9 +85,14 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);#define MFILL_ATOMIC_FLAG (nr) ((__force uffd_flags_t) MFILL_ATOMIC_BIT(nr))#define MFILL_ATOMIC_MODE_MASK ((__force uffd_flags_t) (MFILL_ATOMIC_BIT(0) - 1)) +static inline enum mfill_atomic_mode uffd_flags_get_mode (uffd_flags_t flags) +{ +return (enum mfill_atomic_mode )(flags &MFILL_ATOMIC_MODE_MASK ); +} +static inline bool uffd_flags_mode_is (uffd_flags_t flags,enum mfill_atomic_mode expected) { -return (flags &MFILL_ATOMIC_MODE_MASK ) == ((__force uffd_flags_t ) expected); +return uffd_flags_get_mode (flags) == expected; }static inline uffd_flags_t uffd_flags_set_mode (uffd_flags_t flags,enum mfill_atomic_mode mode)@@ -196,41 +201,16 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -unsigned long vm_flags, -bool wp_async) +static inline const vm_uffd_ops *vma_get_uffd_ops (struct vm_area_struct *vma) { - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true;
include/linux/userfaultfd_k.h static inline bool userfaultfd_armed(struct vm_area_struct *vma) · line 227
Quoted 10 lines
include/linux/userfaultfd_k.h +0
-10- -#ifndef CONFIG_PTE_MARKER_UFFD_WP -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) -return false; -#endif
Hi Peter,
Thanks for this cleanup!
It looks like the above two checks, the wp-async one and the PTE marker check, have been reordered in this patch. Does this result in a functional difference?
The rest of this series looks fine to me. :)
Quoted 206 lines
include/linux/userfaultfd_k.h +6
-4- -/* By default, allow any of anon|shmem|hugetlb */ -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); +if (vma->vm_ops && vma->vm_ops->userfaultfd_ops) +return vma->vm_ops->userfaultfd_ops; +return NULL ; } +bool vma_can_userfault (struct vm_area_struct *vma, +unsigned long vm_flags,bool wp_async); +static inline bool vma_has_uffd_without_event_remap (struct vm_area_struct *vma) {struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;mm/shmem.c +1
-1@@ -3158,7 +3158,7 @@ static int shmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff,return shmem_get_folio (inode, pgoff,0 , folio,SGP_NOALLOC ); } -int shmem_mfill_atomic_pte(pmd_t *dst_pmd, +static int shmem_mfill_atomic_pte(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr,mm/userfaultfd.c +87
-29@@ -14,12 +14,48 @@#include <linux/userfaultfd_k.h> #include <linux/mmu_notifier.h> #include <linux/hugetlb.h> -#include <linux/shmem_fs.h> #include <asm/tlbflush.h> #include <asm/tlb.h> #include "internal.h" #include "swap.h" +bool vma_can_userfault (struct vm_area_struct *vma, +unsigned long vm_flags,bool wp_async) +{ +unsigned long supported; + +if (vma->vm_flags &VM_DROPPABLE ) +return false; + + vm_flags &= __VM_UFFD_FLAGS; + +#ifndef CONFIG_PTE_MARKER_UFFD_WP +/* +* If user requested uffd-wp but not enabled pte markers for +* uffd-wp, then any file system (like shmem or hugetlbfs) are not +* supported but only anonymous. +*/ +if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) +return false; +#endif +/* +* If wp async enabled, and WP is the only mode enabled, allow any +* memory type. +*/ +if (wp_async && (vm_flags ==VM_UFFD_WP )) +return true; + +if (vma_is_anonymous (vma)) +/* Anonymous has no page cache, MINOR not supported */ + supported =VM_UFFD_MISSING |VM_UFFD_WP ; +else if (vma_get_uffd_ops (vma)) + supported =vma_get_uffd_ops (vma)->uffd_features; +else +return false; + +return !(vm_flags & (~supported)); +} +static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) {@@ -384,11 +420,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, {struct inode *inode =file_inode (dst_vma->vm_file);pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +const vm_uffd_ops *uffd_ops =vma_get_uffd_ops (dst_vma);struct folio *folio;struct page *page;int ret; - ret =shmem_get_folio (inode, pgoff,0 , &folio,SGP_NOALLOC ); +if (WARN_ON_ONCE (!uffd_ops || !uffd_ops->uffd_get_folio)) +return -EINVAL ; + + ret = uffd_ops->uffd_get_folio (inode, pgoff, &folio);/* Our caller expects us to return -EFAULT if we failed to find folio */ if (ret == -ENOENT ) ret = -EFAULT ;@@ -504,18 +544,6 @@ static __always_inline ssize_t mfill_atomic_hugetlb(u32 hash;struct address_space *mapping; -/* -* There is no default zero huge page for all huge page sizes as -* supported by hugetlb. A PMD_SIZE huge pages may exist as used -* by THP. Since we can not reliably insert a zero page, this -* feature is not supported. -*/ -if (uffd_flags_mode_is (flags,MFILL_ATOMIC_ZEROPAGE )) { -up_read (&ctx->map_changing_lock); -uffd_mfill_unlock (dst_vma); -return -EINVAL ; - } - src_addr = src_start; dst_addr = dst_start; copied =0 ;@@ -686,14 +714,55 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, err =mfill_atomic_pte_zeropage (dst_pmd, dst_vma, dst_addr); }else { - err =shmem_mfill_atomic_pte (dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); +const vm_uffd_ops *uffd_ops =vma_get_uffd_ops (dst_vma); + +if (WARN_ON_ONCE (!uffd_ops || !uffd_ops->uffd_copy)) { + err = -EINVAL ; + }else { + err = uffd_ops->uffd_copy (dst_pmd, dst_vma, + dst_addr, src_addr, + flags, foliop); + } }return err; } +static inline bool +vma_uffd_ops_supported (struct vm_area_struct *vma,uffd_flags_t flags) +{ +enum mfill_atomic_mode mode =uffd_flags_get_mode (flags); +const vm_uffd_ops *uffd_ops; +unsigned long uffd_ioctls; + +if ((flags &MFILL_ATOMIC_WP ) && !(vma->vm_flags &VM_UFFD_WP )) +return false; + +/* Anonymous supports everything except CONTINUE */ +if (vma_is_anonymous (vma)) +return mode !=MFILL_ATOMIC_CONTINUE ; + + uffd_ops =vma_get_uffd_ops (vma); +if (!uffd_ops) +return false; + + uffd_ioctls = uffd_ops->uffd_ioctls; +switch (mode) { +case MFILL_ATOMIC_COPY : +return uffd_ioctls &BIT (_UFFDIO_COPY); +case MFILL_ATOMIC_ZEROPAGE : +return uffd_ioctls &BIT (_UFFDIO_ZEROPAGE); +case MFILL_ATOMIC_CONTINUE : +if (!(vma->vm_flags &VM_SHARED )) +return false; +return uffd_ioctls &BIT (_UFFDIO_CONTINUE); +case MFILL_ATOMIC_POISON : +return uffd_ioctls &BIT (_UFFDIO_POISON); +default : +return false; + } +} + static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long dst_start, unsigned long src_start,@@ -752,11 +821,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, dst_vma->vm_flags & VM_SHARED)) goto out_unlock; -/* -* validate 'mode' now that we know the dst_vma: don't allow -* a wrprotect copy if the userfaultfd didn't register as WP. -*/ -if ((flags &MFILL_ATOMIC_WP ) && !(dst_vma->vm_flags &VM_UFFD_WP )) +if (!vma_uffd_ops_supported (dst_vma, flags)) goto out_unlock;/* @@ -766,12 +831,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,return mfill_atomic_hugetlb (ctx, dst_vma, dst_start, src_start, len, flags); -if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) - goto out_unlock; -if (!vma_is_shmem (dst_vma) && -uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE )) - goto out_unlock; -while (src_addr < src_start + len) {pmd_t dst_pmdval; --2.49.0
Peter Xu
reviewerRe: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm
Quoted 33 lines · James Houghton
Quoted 20 lines
include/linux/userfaultfd_k.h +1
-18-static inline bool vma_can_userfault (struct vm_area_struct *vma, -unsigned long vm_flags, -bool wp_async) +static inline const vm_uffd_ops *vma_get_uffd_ops (struct vm_area_struct *vma) { - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true;include/linux/userfaultfd_k.h static inline bool userfaultfd_armed(struct vm_area_struct *vma) · line 227
Quoted 10 lines
include/linux/userfaultfd_k.h +0
-10- -#ifndef CONFIG_PTE_MARKER_UFFD_WP -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) -return false; -#endif Hi Peter,
Quoted 7 lines
Thanks for this cleanup!
It looks like the above two checks, the wp-async one and the PTE marker check, have been reordered in this patch. Does this result in a functional difference?
The rest of this series looks fine to me. :)
Thanks for the very careful review, James!
Yes that's a small tweak I did when moving. I don't expect to have any functional change. Maybe I should at least mention that in the commit log.
Here I did the movement because fundamentally wp_async depends on the pte markers, so it may be slightly more intuitive to check pte markers first, rejecting any form of file wr-protect traps. Otherwise it may looks like we could return the true for wp_async==true too early. In reality IIUC it can't happen.
For example, currently userfaultfd_api() has:
#ifndef CONFIG_PTE_MARKER_UFFD_WP uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM; uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED; uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
#endif
So when wp_async can be true above, pte markers must be compiled.. IOW, above code clip should work identically with below lines:
#ifdef CONFIG_PTE_MARKER_UFFD_WP
if (wp_async && (vm_flags == VM_UFFD_WP))
return true;
#endif
#ifndef CONFIG_PTE_MARKER_UFFD_WP
if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))
return false;
#endifThen it means both chunks of code cannot be compiled together. The order shouldn't matter.
But maybe I should just move it back as before, to save the explain and confusions. Let me know if you have any preference.
Thanks,
Signature · 1 line
Peter Xu
James Houghton
Reviewed-byRe: [PATCH 4/4] mm: Apply vm_uffd_ops API to core mm
Quoted 41 lines · Peter Xu
Quoted 39 lines · James Houghton
Quoted 20 lines
include/linux/userfaultfd_k.h +1
-18-static inline bool vma_can_userfault (struct vm_area_struct *vma, -unsigned long vm_flags, -bool wp_async) +static inline const vm_uffd_ops *vma_get_uffd_ops (struct vm_area_struct *vma) { - vm_flags &= __VM_UFFD_FLAGS; - -if (vma->vm_flags &VM_DROPPABLE ) -return false; - -if ((vm_flags &VM_UFFD_MINOR ) && - (!is_vm_hugetlb_page (vma) && !vma_is_shmem (vma))) -return false; - -/* -* If wp async enabled, and WP is the only mode enabled, allow any -* memory type. -*/ -if (wp_async && (vm_flags ==VM_UFFD_WP )) -return true;include/linux/userfaultfd_k.h static inline bool userfaultfd_armed(struct vm_area_struct *vma) · line 227
Quoted 10 lines
include/linux/userfaultfd_k.h +0
-10- -#ifndef CONFIG_PTE_MARKER_UFFD_WP -/* -* If user requested uffd-wp but not enabled pte markers for -* uffd-wp, then shmem & hugetlbfs are not supported but only -* anonymous. -*/ -if ((vm_flags &VM_UFFD_WP ) && !vma_is_anonymous (vma)) -return false; -#endif Hi Peter,
Thanks for this cleanup!
It looks like the above two checks, the wp-async one and the PTE marker check, have been reordered in this patch. Does this result in a functional difference?
Quoted 6 lines
Quoted 1 line
The rest of this series looks fine to me. :)
Thanks for the very careful review, James!
Yes that's a small tweak I did when moving. I don't expect to have any functional change. Maybe I should at least mention that in the commit log.
Yeah if you could leave a small explanation, like what you have below, in the commit log, that would be good. :)
Quoted 7 lines
Here I did the movement because fundamentally
wp_asyncdepends on the pte markers, so it may be slightly more intuitive to check pte markers first, rejecting any form of file wr-protect traps. Otherwise it may looks like we could return the true for wp_async==true too early. In reality IIUC it can't happen.
Quoted 7 lines
For example, currently
userfaultfd_api()has:#ifndef CONFIG_PTE_MARKER_UFFD_WP uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM; uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED; uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;#endif
Ah, I see, thanks!
Quoted 13 lines
So when
wp_asynccan be true above, pte markers must be compiled.. IOW, above code clip should work identically with below lines:#ifdef CONFIG_PTE_MARKER_UFFD_WP if (wp_async && (vm_flags == VM_UFFD_WP)) return true; #endif #ifndef CONFIG_PTE_MARKER_UFFD_WP if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) return false; #endif
Quoted 5 lines
Then it means both chunks of code cannot be compiled together. The order shouldn't matter.
But maybe I should just move it back as before, to save the explain and confusions. Let me know if you have any preference.
Feel free to leave this patch as it is, that's fine. Thanks for the explanation. :)
If you'd like, feel free to add:
Quoted 8 lines · Peter Xu
[based on akpm/mm-new]
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Thanks for posting this. I confirmed that minor fault handling was working for guest_memfd based on this series and looked simple (a draft based on mmap support in guest_memfd v7 [1]):
virt/kvm/guest_memfd.c +42 -0
@@ -5,6 +5,9 @@#include <linux/pagemap.h> #include <linux/anon_inodes.h> #include <linux/set_memory.h> +#ifdef CONFIG_USERFAULTFD +#include <linux/userfaultfd_k.h> +#endif #include "kvm_mm.h"
@@ -396,6 +399,14 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)kvm_gmem_mark_prepared (folio); } +#ifdef CONFIG_USERFAULTFD +if (userfaultfd_minor (vmf->vma)) { +folio_unlock (folio); +filemap_invalidate_unlock_shared (inode->i_mapping); +return handle_userfault (vmf,VM_UFFD_MINOR ); + } +#endif + vmf->page =folio_file_page (folio, vmf->pgoff); out_folio:
@@ -410,8 +421,39 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)return ret; } +#ifdef CONFIG_USERFAULTFD +static int kvm_gmem_uffd_get_folio (struct inode *inode,pgoff_t pgoff, +struct folio **foliop) +{ +struct folio *folio; + folio =kvm_gmem_get_folio (inode, pgoff); + +if (IS_ERR (folio)) { + *foliop =NULL ; +return PTR_ERR (folio); + } + +if (!folio_test_uptodate (folio)) { +clear_highpage (folio_page (folio,0 )); +kvm_gmem_mark_prepared (folio); + } + + *foliop = folio; +return 0 ; +} + +static const vm_uffd_ops kvm_gmem_uffd_ops = { + .uffd_features =VM_UFFD_MINOR , + .uffd_ioctls =BIT (_UFFDIO_CONTINUE), + .uffd_get_folio = kvm_gmem_uffd_get_folio, +}; +#endif +static const struct vm_operations_struct kvm_gmem_vm_ops = { .fault = kvm_gmem_fault, +#ifdef CONFIG_USERFAULTFD + .userfaultfd_ops = &kvm_gmem_uffd_ops, +#endif };static int kvm_gmem_mmap (struct file *file,struct vm_area_struct *vma)
[1]: https://lore.kernel.org/kvm/[email protected]/
Quoted 73 lines
Here, the major goal is to make kernel modules be able to opt-in with any form of userfaultfd supports, like guest-memfd. This alternative option should hopefully be cleaner, and avoid leaking userfault details into
vm_ops.fault().It also means this series does not depend on anything. It's a pure refactoring of userfaultfd internals to provide a generic API, so that other types of files, especially RAM based, can support userfaultfd without touching
mm/at all.To achieve it, this series introduced a file operation called
vm_uffd_ops. The ops needs to be provided when a file type supports any of userfaultfd.With that, I moved both hugetlbfs and shmem over.
Hugetlbfs is still very special that it will only use partial of the
vm_uffd_opsAPI, due to similar reason whyhugetlb_vm_op_fault()has aBUG()and so far hard-coded into core mm. But this should still be better, because at least hugetlbfs is still always involved in feature probing (e.g. where it used to not support ZEROPAGE and we have a hard-coded line to fail that, and some more). Meanwhile after this series, shmem will be completely converted to the newvm_uffd_opsAPI; the finalvm_uffd_opsfor shmem looks like this:static const
vm_uffd_opsshmem_uffd_ops= {.uffd_features = __VM_UFFD_FLAGS, .uffd_ioctls = BIT(_UFFDIO_COPY) |
BIT(_UFFDIO_ZEROPAGE)|BIT(_UFFDIO_WRITEPROTECT)|BIT(_UFFDIO_CONTINUE)|BIT(_UFFDIO_POISON),
.uffd_get_folio =shmem_uffd_get_folio,
.uffd_copy =shmem_mfill_atomic_pte,
};As I mentioned in one of my reply to Nikita, I don't like the current interface of
uffd_copy(), but this will be the minimum change version of such API to support complete extrenal-module-ready userfaultfd. Here, very minimal change will be needed from shmem side to support that.Meanwhile, the
vm_uffd_opsis also not the only place one will need to provide to support userfaultfd. Normallyvm_ops.fault()will also need to be updated, but that's a generic function and it'll play together with the newvm_uffd_opsto make everything fly.No functional change expected at all after the whole series applied. There might be some slightly stricter check on uffd ops here and there in the last patch, but that really shouldn't stand out anywhere to anyone.
For testing: besides the cross-compilation tests, I did also try with uffd-stress in a VM to measure any perf difference before/after the change; The static call becomes a pointer now. I really cannot measure anything different, which is more or less expected.
Comments welcomed, thanks.
Peter Xu (4):
mm: Introducevm_uffd_opsAPI
mm/shmem: Supportvm_uffd_opsAPI
mm/hugetlb: Supportvm_uffd_opsAPI
mm: Applyvm_uffd_opsAPI to core mminclude/linux/mm.h | 71 +++++++++++++++++++++ include/linux/shmem_fs.h | 14 ----- include/linux/userfaultfd_k.h | 58 ++++------------- mm/hugetlb.c | 19 ++++++ mm/shmem.c | 28 ++++++++- mm/userfaultfd.c | 115 +++++++++++++++++++++++++--------- 6 files changed, 217 insertions(+), 88 deletions(-)--
2.49.0
Peter Xu
reviewerQuoted 5 lines · Nikita Kalyazin
Quoted 2 lines · Peter Xu
[based on akpm/mm-new]
Quoted 8 lines
Quoted 6 lines
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Hi, Nikita,
Quoted 4 lines
Thanks for posting this. I confirmed that minor fault handling was working for
guest_memfdbased on this series and looked simple (a draft based on mmap support inguest_memfdv7 [1]):
Thanks for the quick spin, glad to know it works. Some trivial things to mention below..
virt/kvm/guest_memfd.c line 8
Quoted 10 lines
virt/kvm/guest_memfd.c +1
-0@@ -5,6 +5,9 @@#include <linux/pagemap.h> #include <linux/anon_inodes.h> #include <linux/set_memory.h> +#ifdef CONFIG_USERFAULTFD
This ifdef not needed, userfaultfd_k.h has taken care of all cases.
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf) · line 402
Quoted 10 lines
virt/kvm/guest_memfd.c +3
-0+#include <linux/userfaultfd_k.h> +#endif #include "kvm_mm.h" @@ -396,6 +399,14 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)kvm_gmem_mark_prepared (folio); } +#ifdef CONFIG_USERFAULTFD
Same here. userfaultfd_minor() is always defined.
I'll wait for a few more days for reviewers, and likely send v2 before next week.
Thanks,
Signature · 1 line
Peter Xu
Quoted 18 lines · Peter Xu
Quoted 5 lines · Nikita Kalyazin
Quoted 2 lines · Peter Xu
[based on akpm/mm-new]
Quoted 8 lines
Quoted 6 lines
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Hi, Nikita,
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Thanks for posting this. I confirmed that minor fault handling was working for
guest_memfdbased on this series and looked simple (a draft based on mmap support inguest_memfdv7 [1]):Thanks for the quick spin, glad to know it works. Some trivial things to mention below..
Following up, I drafted UFFDIO_COPY support for guest_memfd to confirm it works as well:
virt/kvm/guest_memfd.c +56 -2
@@ -349,12 +349,19 @@ static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)static vm_fault_t kvm_gmem_fault (struct vm_fault *vmf) { +struct vm_area_struct *vma = vmf ? vmf->vma :NULL ;struct inode *inode =file_inode (vmf->vma->vm_file);struct folio *folio;vm_fault_t ret =VM_FAULT_LOCKED ;filemap_invalidate_lock_shared (inode->i_mapping); + folio =filemap_get_entry (inode->i_mapping, vmf->pgoff); +if (!folio && vma &&userfaultfd_missing (vma)) { +filemap_invalidate_unlock_shared (inode->i_mapping); +return handle_userfault (vmf,VM_UFFD_MISSING ); + } + folio =kvm_gmem_get_folio (inode, vmf->pgoff);if (IS_ERR (folio)) {int err =PTR_ERR (folio);
@@ -438,10 +445,57 @@ static int kvm_gmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff,return 0 ; } +static int kvm_gmem_mfill_atomic_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr, +unsigned long src_addr, +uffd_flags_t flags, +struct folio **foliop) +{ +struct inode *inode =file_inode (dst_vma->vm_file); +pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct folio *folio; +int ret; + + folio =kvm_gmem_get_folio (inode, pgoff); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); + goto out; + } + +folio_unlock (folio); + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { +void *vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, (const void __user *)src_addr,PAGE_SIZE ); +kunmap_local (vaddr); +if (unlikely (ret)) { + *foliop = folio; + ret = -ENOENT ; + goto out; + } + }else {/* ZEROPAGE */ +clear_user_highpage (&folio->page, dst_addr); + } + +kvm_gmem_mark_prepared (folio); + + ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + &folio->page, true, flags); + +if (ret) +folio_put (folio); +out: +return ret; +} +static const vm_uffd_ops kvm_gmem_uffd_ops = { - .uffd_features =VM_UFFD_MINOR , - .uffd_ioctls =BIT (_UFFDIO_CONTINUE), + .uffd_features =VM_UFFD_MISSING |VM_UFFD_MINOR , + .uffd_ioctls =BIT (_UFFDIO_COPY) | +BIT (_UFFDIO_ZEROPAGE) | +BIT (_UFFDIO_CONTINUE), .uffd_get_folio = kvm_gmem_uffd_get_folio, + .uffd_copy = kvm_gmem_mfill_atomic_pte, };#endif
Quoted 13 lines
virt/kvm/guest_memfd.c line 8
Quoted 11 lines
virt/kvm/guest_memfd.c +1
-0@@ -5,6 +5,9 @@#include <linux/pagemap.h> #include <linux/anon_inodes.h> #include <linux/set_memory.h> +#ifdef CONFIG_USERFAULTFD This ifdef not needed,
userfaultfd_k.hhas taken care of all cases.
Good to know, thanks.
Quoted 5 lines
Quoted 5 lines
virt/kvm/guest_memfd.c +2
-0+#include <linux/userfaultfd_k.h> +#endif #include "kvm_mm.h"
Quoted 7 lines
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf) · line 402
Quoted 5 lines
virt/kvm/guest_memfd.c +1
-0@@ -396,6 +399,14 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)kvm_gmem_mark_prepared (folio); } +#ifdef CONFIG_USERFAULTFD Same here.
userfaultfd_minor()is always defined.
Thank you.
Quoted 8 lines
I'll wait for a few more days for reviewers, and likely send v2 before next week.
Thanks,
--
Peter Xu
Peter Xu
reviewerQuoted 25 lines · Nikita Kalyazin
Quoted 22 lines · Peter Xu
Quoted 5 lines · Nikita Kalyazin
Quoted 2 lines · Peter Xu
[based on akpm/mm-new]
Quoted 8 lines
Quoted 6 lines
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Hi, Nikita,
Quoted 4 lines
Thanks for posting this. I confirmed that minor fault handling was working for
guest_memfdbased on this series and looked simple (a draft based on mmap support inguest_memfdv7 [1]):
Quoted 5 lines
Quoted 2 lines
Thanks for the quick spin, glad to know it works. Some trivial things to mention below..
Following up, I drafted
UFFDIO_COPYsupport forguest_memfdto confirm it works as well:
Appreciated.
Since at it, I'll comment quickly below.
Quoted 12 lines
virt/kvm/guest_memfd.c +1
-0@@ -349,12 +349,19 @@ static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)static vm_fault_t kvm_gmem_fault (struct vm_fault *vmf) { +struct vm_area_struct *vma = vmf ? vmf->vma :NULL ;struct inode *inode =file_inode (vmf->vma->vm_file);
virt/kvm/guest_memfd.c static bool kvm_gmem_offset_is_shared(struct file · line 364
Quoted 10 lines
virt/kvm/guest_memfd.c +5
-0struct folio *folio;vm_fault_t ret =VM_FAULT_LOCKED ;filemap_invalidate_lock_shared (inode->i_mapping); + folio =filemap_get_entry (inode->i_mapping, vmf->pgoff); +if (!folio && vma &&userfaultfd_missing (vma)) { +filemap_invalidate_unlock_shared (inode->i_mapping); +return handle_userfault (vmf,VM_UFFD_MISSING ); + }
Likely a possible refcount leak when folio != NULL here.
Quoted 33 lines
virt/kvm/guest_memfd.c +1
-0+ folio =kvm_gmem_get_folio (inode, vmf->pgoff);if (IS_ERR (folio)) {virt/kvm/guest_memfd.c +24
-0int err =PTR_ERR (folio);@@ -438,10 +445,57 @@ static int kvm_gmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff, return 0; } +static int kvm_gmem_mfill_atomic_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr, +unsigned long src_addr, +uffd_flags_t flags, +struct folio **foliop) +{ +struct inode *inode =file_inode (dst_vma->vm_file); +pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct folio *folio; +int ret; + + folio =kvm_gmem_get_folio (inode, pgoff); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); + goto out; + } + +folio_unlock (folio); + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { +void *vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, (const void __user *)src_addr,PAGE_SIZE ); +kunmap_local (vaddr);
virt/kvm/guest_memfd.c static int kvm_gmem_uffd_get_folio(struct inode · line 482
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+if (unlikely (ret)) { + *foliop = folio; + ret = -ENOENT ; + goto out; + } + }else {/* ZEROPAGE */ +clear_user_highpage (&folio->page, dst_addr); + } + +kvm_gmem_mark_prepared (folio);
Since Faud's series hasn't yet landed, so I'm almost looking at the current code base with an imagination of what might happen.
In general, missing trapping for guest-memfd could start to be slightly trickier. So far IIUC guest-memfd cache pool needs to be populated only by a prior fallocate() syscall, not during fault. So I suppose we will need to use uptodate bit to mark folio ready, like what's done here.
If so, we may want to make sure in fault path any !uptodate fault will get trapped for missing too, even if it sounds not strictly a "cache miss" ... so slightly confusing but sounds necessary.
Meanwhile, I'm not 100% sure how it goes especially if taking CoCo into account, because CoCo needs to prepare the pages, so mark uptodate may not be enough? I don't know well on the CoCo side to tell. Otherwise we'll at least need to restrict MISSING traps to only happen on fully shared guest-memfds.
OTOH, MINOR should be much easier to be done for guest-memfd, not only because the code to support that would be very minimum which is definitely lovely, but also because it's still pretty common idea to monitor pgtable entries, and it should logically even apply to CoCo: in a fault(), we need to check whether the guest-memfd folio is "shared" and/or "faultable" first; it should already fail the fault() if it's a private folio. Then if it's visible (aka, "faultable") to HVA namespace, then it's legal to trap a MINOR too. For !CoCo it'll always trap as it's always faultable.
MINOR also makes more sense to be used in the future with 1G postcopy support on top of gmem, because that's almost the only way to go. Looks like we've made up our mind to reuse Hugetlb pages for gmem which sounds good, then Hugetlb pages are in 1G granule in allocations, and we can't easily do 4K miss trapping on one 1G huge page. MINOR is simpler but actually more powerful from that POV.
To summarize, I think after this we can do MINOR before MISSING for guest-memfd if MINOR already works for you. We can leave MISSING until we know how we would use it.
Thanks,
Quoted 62 lines
virt/kvm/guest_memfd.c +15
-2+ + ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + &folio->page, true, flags); + +if (ret) +folio_put (folio); +out: + return ret; +} +static const vm_uffd_ops kvm_gmem_uffd_ops = { - .uffd_features =VM_UFFD_MINOR , - .uffd_ioctls =BIT (_UFFDIO_CONTINUE), + .uffd_features =VM_UFFD_MISSING |VM_UFFD_MINOR , + .uffd_ioctls =BIT (_UFFDIO_COPY) | +BIT (_UFFDIO_ZEROPAGE) | +BIT (_UFFDIO_CONTINUE), .uffd_get_folio = kvm_gmem_uffd_get_folio, + .uffd_copy = kvm_gmem_mfill_atomic_pte, };#endif Quoted 14 lines
virt/kvm/guest_memfd.c line 8
Quoted 12 lines
virt/kvm/guest_memfd.c +1
-0@@ -5,6 +5,9 @@#include <linux/pagemap.h> #include <linux/anon_inodes.h> #include <linux/set_memory.h> +#ifdef CONFIG_USERFAULTFD This ifdef not needed,
userfaultfd_k.hhas taken care of all cases.Good to know, thanks.
Quoted 5 lines
Quoted 5 lines
virt/kvm/guest_memfd.c +2
-0+#include <linux/userfaultfd_k.h> +#endif #include "kvm_mm.h" Quoted 7 lines
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf) · line 402
Quoted 5 lines
virt/kvm/guest_memfd.c +1
-0@@ -396,6 +399,14 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)kvm_gmem_mark_prepared (folio); } +#ifdef CONFIG_USERFAULTFD Same here.
userfaultfd_minor()is always defined.Thank you.
Quoted 8 lines
I'll wait for a few more days for reviewers, and likely send v2 before next week.
Thanks,
--
Peter Xu
Signature · 1 line
Peter Xu
Quoted 53 lines · Peter Xu
Quoted 25 lines · Nikita Kalyazin
Quoted 22 lines · Peter Xu
Quoted 5 lines · Nikita Kalyazin
Quoted 2 lines · Peter Xu
[based on akpm/mm-new]
Quoted 8 lines
Quoted 6 lines
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Hi, Nikita,
Quoted 4 lines
Thanks for posting this. I confirmed that minor fault handling was working for
guest_memfdbased on this series and looked simple (a draft based on mmap support inguest_memfdv7 [1]):Quoted 5 lines
Quoted 2 lines
Thanks for the quick spin, glad to know it works. Some trivial things to mention below..
Following up, I drafted
UFFDIO_COPYsupport forguest_memfdto confirm it works as well:Appreciated.
Since at it, I'll comment quickly below.
Quoted 17 lines
virt/kvm/guest_memfd.c +1
-0@@ -349,12 +349,19 @@ static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)static vm_fault_t kvm_gmem_fault (struct vm_fault *vmf) { +struct vm_area_struct *vma = vmf ? vmf->vma :NULL ;struct inode *inode =file_inode (vmf->vma->vm_file);struct folio *folio;vm_fault_t ret =VM_FAULT_LOCKED ;filemap_invalidate_lock_shared (inode->i_mapping);
Quoted 7 lines
virt/kvm/guest_memfd.c static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index) · line 363
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ folio =filemap_get_entry (inode->i_mapping, vmf->pgoff); +if (!folio && vma &&userfaultfd_missing (vma)) { +filemap_invalidate_unlock_shared (inode->i_mapping); +return handle_userfault (vmf,VM_UFFD_MISSING ); + }Likely a possible refcount leak when folio != NULL here.
Thank you. I was only aiming to cover the happy case for know. I will keep it in mind for the future.
Quoted 47 lines
Quoted 33 lines
virt/kvm/guest_memfd.c +1
-0+ folio =kvm_gmem_get_folio (inode, vmf->pgoff);if (IS_ERR (folio)) {int err = PTR_ERR(folio);virt/kvm/guest_memfd.c +24
-0@@ -438,10 +445,57 @@ static int kvm_gmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff, return 0; } +static int kvm_gmem_mfill_atomic_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr, +unsigned long src_addr, +uffd_flags_t flags, +struct folio **foliop) +{ +struct inode *inode =file_inode (dst_vma->vm_file); +pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct folio *folio; +int ret; + + folio =kvm_gmem_get_folio (inode, pgoff); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); + goto out; + } + +folio_unlock (folio); + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { +void *vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, (const void __user *)src_addr,PAGE_SIZE ); +kunmap_local (vaddr);virt/kvm/guest_memfd.c static int kvm_gmem_uffd_get_folio(struct inode · line 482
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+if (unlikely (ret)) { + *foliop = folio; + ret = -ENOENT ; + goto out; + } + }else {/* ZEROPAGE */ +clear_user_highpage (&folio->page, dst_addr); + } + +kvm_gmem_mark_prepared (folio);Since Faud's series hasn't yet landed, so I'm almost looking at the current code base with an imagination of what might happen.
Quoted 4 lines
In general, missing trapping for guest-memfd could start to be slightly trickier. So far IIUC guest-memfd cache pool needs to be populated only by a prior
fallocate()syscall, not during fault. So I suppose we will need to use uptodate bit to mark folio ready, like what's done here.
I don't think I'm familiar with the fallocate() requirement in guest_memfd. Fuad's v12 [1] (although I think it has been like that from the beginning) calls kvm_gmem_get_folio() that populates pagecache in the fault handler (kvm_gmem_fault_shared()). SEV [2] and TDX [3] seem to use kvm_gmem_populate() for both allocation and preparation.
[1] https://lore.kernel.org/kvm/[email protected]/T/#m15b53a741e4f328e61f995a01afb9c4682ffe611
[2] https://elixir.bootlin.com/linux/v6.16-rc3/source/arch/x86/kvm/svm/sev.c#L2331
[3] https://elixir.bootlin.com/linux/v6.16-rc3/source/arch/x86/kvm/vmx/tdx.c#L3236
Quoted 10 lines
If so, we may want to make sure in fault path any !uptodate fault will get trapped for missing too, even if it sounds not strictly a "cache miss" ... so slightly confusing but sounds necessary.
Meanwhile, I'm not 100% sure how it goes especially if taking CoCo into account, because CoCo needs to prepare the pages, so mark uptodate may not be enough? I don't know well on the CoCo side to tell. Otherwise we'll at least need to restrict MISSING traps to only happen on fully shared guest-memfds.
I am not fluent in CoCo either, but I thought CoCo needed to do preparation for private pages only, while UFFD shouldn't be dealing with them so issuing MISSING only on shared looks sensible to me.
Quoted 14 lines
OTOH, MINOR should be much easier to be done for guest-memfd, not only
because the code to support that would be very minimum which is definitely
lovely, but also because it's still pretty common idea to monitor pgtable
entries, and it should logically even apply to CoCo: in afault(), we need
to check whether the guest-memfd folio is "shared" and/or "faultable"
first; it should already fail thefault()if it's a private folio. Then if
it's visible (aka, "faultable") to HVA namespace, then it's legal to trap a
MINOR too. For !CoCo it'll always trap as it's always faultable. > MINOR also makes more sense to be used in the future with 1G postcopy support on top of gmem, because that's almost the only way to go. Looks
like we've made up our mind to reuse Hugetlb pages for gmem which sounds
good, then Hugetlb pages are in 1G granule in allocations, and we can't
easily do 4K miss trapping on one 1G huge page. MINOR is simpler but
actually more powerful from that POV.
Quoted 3 lines
To summarize, I think after this we can do MINOR before MISSING for guest-memfd if MINOR already works for you. We can leave MISSING until we know how we would use it.
Starting with MINOR sounds good to me.
Quoted 69 lines
Thanks,
Quoted 47 lines
virt/kvm/guest_memfd.c +15
-2+ + ret =mfill_atomic_install_pte (dst_pmd, dst_vma, dst_addr, + &folio->page, true, flags); + +if (ret) +folio_put (folio); +out: + return ret; +} +static const vm_uffd_ops kvm_gmem_uffd_ops = { - .uffd_features =VM_UFFD_MINOR , - .uffd_ioctls =BIT (_UFFDIO_CONTINUE), + .uffd_features =VM_UFFD_MISSING |VM_UFFD_MINOR , + .uffd_ioctls =BIT (_UFFDIO_COPY) | +BIT (_UFFDIO_ZEROPAGE) | +BIT (_UFFDIO_CONTINUE), .uffd_get_folio = kvm_gmem_uffd_get_folio, + .uffd_copy = kvm_gmem_mfill_atomic_pte, };#endif Quoted 14 lines
virt/kvm/guest_memfd.c line 8
Quoted 12 lines
virt/kvm/guest_memfd.c +1
-0@@ -5,6 +5,9 @@#include <linux/pagemap.h> #include <linux/anon_inodes.h> #include <linux/set_memory.h> +#ifdef CONFIG_USERFAULTFD This ifdef not needed,
userfaultfd_k.hhas taken care of all cases.Good to know, thanks.
Quoted 9 lines
Quoted 9 lines
virt/kvm/guest_memfd.c +2
-0+#include <linux/userfaultfd_k.h> +#endif #include "kvm_mm.h" @@ -396,6 +399,14 @@ static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf)kvm_gmem_mark_prepared (folio); }Quoted 15 lines
Quoted 3 lines
virt/kvm/guest_memfd.c static vm_fault_t kvm_gmem_fault(struct vm_fault *vmf) · line 402
Quoted 1 line
virt/kvm/guest_memfd.c +1
-0+#ifdef CONFIG_USERFAULTFD Same here.
userfaultfd_minor()is always defined.Thank you.
Quoted 8 lines
I'll wait for a few more days for reviewers, and likely send v2 before next week.
Thanks,
--
Peter Xu--
Peter Xu
Peter Xu
reviewerQuoted 64 lines · Nikita Kalyazin
Quoted 53 lines · Peter Xu
Quoted 25 lines · Nikita Kalyazin
Quoted 22 lines · Peter Xu
Quoted 5 lines · Nikita Kalyazin
Quoted 2 lines · Peter Xu
[based on akpm/mm-new]
Quoted 8 lines
Quoted 6 lines
This series is an alternative proposal of what Nikita proposed here on the initial three patches:
https://lore.kernel.org/r/[email protected]This is not yet relevant to any guest-memfd support, but paving way for it.
Hi Peter,
Hi, Nikita,
Quoted 4 lines
Thanks for posting this. I confirmed that minor fault handling was working for
guest_memfdbased on this series and looked simple (a draft based on mmap support inguest_memfdv7 [1]):Quoted 5 lines
Quoted 2 lines
Thanks for the quick spin, glad to know it works. Some trivial things to mention below..
Following up, I drafted
UFFDIO_COPYsupport forguest_memfdto confirm it works as well:Appreciated.
Since at it, I'll comment quickly below.
Quoted 17 lines
virt/kvm/guest_memfd.c +1
-0@@ -349,12 +349,19 @@ static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)static vm_fault_t kvm_gmem_fault (struct vm_fault *vmf) { +struct vm_area_struct *vma = vmf ? vmf->vma :NULL ;struct inode *inode =file_inode (vmf->vma->vm_file);struct folio *folio;vm_fault_t ret =VM_FAULT_LOCKED ;filemap_invalidate_lock_shared (inode->i_mapping);Quoted 7 lines
virt/kvm/guest_memfd.c static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index) · line 363
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ folio =filemap_get_entry (inode->i_mapping, vmf->pgoff); +if (!folio && vma &&userfaultfd_missing (vma)) { +filemap_invalidate_unlock_shared (inode->i_mapping); +return handle_userfault (vmf,VM_UFFD_MISSING ); + }Likely a possible refcount leak when folio != NULL here.
Quoted 2 lines
Thank you. I was only aiming to cover the happy case for know. I will keep it in mind for the future.
Yep that's good enough, thanks. It's really something I'd comment passingly, it's definitely reassuring to know the happy case works.
Quoted 52 lines
Quoted 47 lines
Quoted 33 lines
virt/kvm/guest_memfd.c +1
-0+ folio =kvm_gmem_get_folio (inode, vmf->pgoff);if (IS_ERR (folio)) {int err = PTR_ERR(folio);virt/kvm/guest_memfd.c +24
-0@@ -438,10 +445,57 @@ static int kvm_gmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff, return 0; } +static int kvm_gmem_mfill_atomic_pte (pmd_t *dst_pmd, +struct vm_area_struct *dst_vma, +unsigned long dst_addr, +unsigned long src_addr, +uffd_flags_t flags, +struct folio **foliop) +{ +struct inode *inode =file_inode (dst_vma->vm_file); +pgoff_t pgoff =linear_page_index (dst_vma, dst_addr); +struct folio *folio; +int ret; + + folio =kvm_gmem_get_folio (inode, pgoff); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); + goto out; + } + +folio_unlock (folio); + +if (uffd_flags_mode_is (flags,MFILL_ATOMIC_COPY )) { +void *vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, (const void __user *)src_addr,PAGE_SIZE ); +kunmap_local (vaddr);virt/kvm/guest_memfd.c static int kvm_gmem_uffd_get_folio(struct inode · line 482
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+if (unlikely (ret)) { + *foliop = folio; + ret = -ENOENT ; + goto out; + } + }else {/* ZEROPAGE */ +clear_user_highpage (&folio->page, dst_addr); + } + +kvm_gmem_mark_prepared (folio);Since Faud's series hasn't yet landed, so I'm almost looking at the current code base with an imagination of what might happen.
Quoted 4 lines
In general, missing trapping for guest-memfd could start to be slightly trickier. So far IIUC guest-memfd cache pool needs to be populated only by a prior
fallocate()syscall, not during fault. So I suppose we will need to use uptodate bit to mark folio ready, like what's done here.
Quoted 5 lines
I don't think I'm familiar with the
fallocate()requirement inguest_memfd. Fuad's v12 [1] (although I think it has been like that from the beginning) callskvm_gmem_get_folio()that populates pagecache in the fault handler (kvm_gmem_fault_shared()). SEV [2] and TDX [3] seem to usekvm_gmem_populate()for both allocation and preparation.
I actually didn't notice fault() uses kvm_gmem_get_folio(), which has FGP_CREAT indeed.
I checked Ackerley's latest 1G patchset, which also did the same that kvm_gmem_get_folio() will invoke the custom allocator to allocate 1G pages even during a fault().
Not sure whether it's intentional though, for example, if the tests in userspace always does fallocate() then the code should run the same, and FGP_CREAT will just never be used.
Thanks for pointing this out. I definitely didn't notice this trivial detail before. Looks like it's not a major issue, if the folio can be dynamically allocated, then MISSING mode (if/when it'll be supported) can capture both "!folio" and "folio && !uptodate" cases here as missing.
Quoted 4 lines
[1] https://lore.kernel.org/kvm/[email protected]/T/#m15b53a741e4f328e61f995a01afb9c4682ffe611
[2] https://elixir.bootlin.com/linux/v6.16-rc3/source/arch/x86/kvm/svm/sev.c#L2331
[3] https://elixir.bootlin.com/linux/v6.16-rc3/source/arch/x86/kvm/vmx/tdx.c#L3236
Signature · 1 line
Peter Xu