guest_memfd: In-place conversion support
Subsystems (10)
v12 cover guest_memfd: In-place conversion support
Here's v12. Thanks everyone for the comments and fast responses! We're now at ~4 weeks to soft-close at 7.3-rc5.
v12 is based on 7.3-rc1 (contains the mm/ bugfixes). Like v11, v12 is also dependent on another series [3], which makes kvm_gmem_get_pfn() NOT return a refcounted page to KVM.
Here's everything stitched together for your convenience:
https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v12
This revision resolves the issue on v10/v11 to do with over-zapping of non-gmem memory because of the requested filters. [1]
The resolution is in a new patch "KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled". I also updated the documentation as requested.
Documentation updates are in these patches:
+ KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
+ KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
+ KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
The documentation update in v10's "KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes" was misplaced imo so I folded it conceptually into the patches above.
private_mem_conversions_test was updated in v11 to test dual backing, and that's meaningless now since with gmem_in_place_conversions, we're now always faulting from guest_memfd, and the userspace_addr not being from the gmem associated with the memslot is considered a user error.
I now only have 1 patch "KVM: selftests: Update private_mem_conversions_test for in-place conversions" instead of 2 in v11.
Resolves discussion at [2].
Here's v12 with tests:
Tested with both CONFIG_KVM_VM_MEMORY_ATTRIBUTES enabled and disabled:
+ tools/testing/selftests/kvm/guest_memfd_test.c
+ tools/testing/selftests/kvm/pre_fault_memory_test.c
+ tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
v11: https://patch.msgid.link/[email protected]
v10: https://lore.kernel.org/r/[email protected]
v9: https://lore.kernel.org/r/[email protected]
v8: https://lore.kernel.org/r/[email protected]
v7: https://lore.kernel.org/r/[email protected]
v6: https://lore.kernel.org/r/[email protected]
RFC v5: https://lore.kernel.org/r/[email protected]
RFC v4: https://lore.kernel.org/all/[email protected]/T/
RFC v3: https://lore.kernel.org/r/[email protected]/T/
RFC v2: https://lore.kernel.org/all/[email protected]/T/
RFC v1: https://lore.kernel.org/all/[email protected]/T/
Previous versions of this feature, part of other series, are available at:
+ https://lore.kernel.org/all/bd163de3118b626d1005aa88e71ef2fb72f0be0f.1726009989.git.ackerleytng@google.com/
+ https://lore.kernel.org/all/[email protected]/
+ https://lore.kernel.org/all/b784326e9ccae6a08388f1bf39db70a2204bdc51.1747264138.git.ackerleytng@google.com/
---
Ackerley Tng (23):
KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
KVM: guest_memfd: Pass mapping type filter to invalidation helper
KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
KVM: guest_memfd: Ensure pages are not in use before conversion
KVM: guest_memfd: Call arch make_shared callback for to-shared conversion
KVM: guest_memfd: Return early if range already has requested attributes
KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
KVM: guest_memfd: Zero page while getting pfn
KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION
KVM: selftests: Test basic single-page conversion flow
KVM: selftests: Test conversion flow when INIT_SHARED
KVM: selftests: Test conversion precision in guest_memfd
KVM: selftests: Test conversion before allocation
KVM: selftests: Convert with allocated folios in different layouts
KVM: selftests: Test that truncation does not change shared/private status
KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
KVM: selftests: Test conversion with elevated page refcount
KVM: selftests: Reset shared memory after hole-punching
KVM: selftests: Provide function to look up guest_memfd details from gpa
KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
KVM: selftests: Set up page size and alignment independently for guest_memfd
KVM: selftests: Update private_mem_conversions_test for in-place conversions
Michael Roth (1):
KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE
Sean Christopherson (21):
KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings
KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES
KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined
KVM: Rename memory attribute APIs to prepare for in-place gmem conversion
KVM: Provide generic interface for checking memory private/shared status
KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h
KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86
KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs
KVM: selftests: Create gmem fd before "regular" fd when adding memslot
KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}
KVM: selftests: Add support for mmap() on guest_memfd in core library
KVM: selftests: Add selftests global for guest memory attributes capability
KVM: selftests: Add helpers for calling ioctls on guest_memfd
KVM: selftests: Test that shared/private status is consistent across processes
KVM: selftests: Provide common function to set memory attributes
KVM: selftests: Update private memory exits test to work with per-gmem attributes
Documentation/admin-guide/kernel-parameters.txt | 25 + Documentation/virt/kvm/api.rst | 107 ++++- .../virt/kvm/x86/amd-memory-encryption.rst | 14 +- Documentation/virt/kvm/x86/intel-tdx.rst | 4 + arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 9 +- arch/x86/kvm/Kconfig | 15 +- arch/x86/kvm/mmu/mmu.c | 28 +- arch/x86/kvm/svm/sev.c | 13 +- arch/x86/kvm/vmx/tdx.c | 8 +- arch/x86/kvm/x86.c | 20 +- include/linux/kvm_host.h | 83 ++-- include/trace/events/kvm.h | 6 +- include/uapi/linux/kvm.h | 16 + mm/folio.c | 2 + tools/testing/selftests/kvm/Makefile.kvm | 1 + tools/testing/selftests/kvm/include/kvm_util.h | 139 +++++- tools/testing/selftests/kvm/include/test_util.h | 34 +- tools/testing/selftests/kvm/lib/kvm_util.c | 222 +++++---- tools/testing/selftests/kvm/lib/test_util.c | 7 - .../kvm/x86/guest_memfd_conversions_test.c | 512 +++++++++++++++++++++ .../kvm/x86/private_mem_conversions_test.c | 66 ++- .../selftests/kvm/x86/private_mem_kvm_exits_test.c | 36 +- virt/kvm/Kconfig | 3 - virt/kvm/guest_memfd.c | 469 +++++++++++++++++-- virt/kvm/kvm_main.c | 92 ++-- 26 files changed, 1646 insertions(+), 287 deletions(-)
---
base-commit: a0c789941a3d705d00c018b3c9348b4b930c0c74
change-id: 20260225-gmem-inplace-conversion-bd0dbd39753a
Best regards,
--
Ackerley Tng <[email protected]>
PATCH v12 1/45 KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
From: Sean Christopherson <[email protected]>
Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleed guest_memfd details into arch code and would be far worse than adding yet another kvm_arch_has... hook.
Opportunistically provide the kvm_arch_gmem_make_private() declaration, and rely on dead-code elimination to eliminate the call to non-existent code when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
--- arch/x86/include/asm/kvm_host.h | 3 +++ include/linux/kvm_host.h | 3 ++- virt/kvm/guest_memfd.c | 5 ++--- 3 files changed, 7 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1854,6 +1854,9 @@ enum kvm_intr_type {#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem (kvm) ((kvm)->arch.has_private_mem)#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert () (!!kvm_x86_ops.gmem_make_private) +#endif #define kvm_arch_has_readonly_mem (kvm) (!(kvm)->arch.has_protected_state)
include/linux/kvm_host.h +2 -1
@@ -2597,9 +2597,10 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, }#endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,kvm_pfn_t nr_pages); +#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert () false#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +2 -3
@@ -772,11 +772,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT -if (kvm_gmem_is_private_mem (file_inode (file), index)) +if (kvm_arch_has_gmem_convert () && +kvm_gmem_is_private_mem (file_inode (file), index)) r =kvm_arch_gmem_make_private (kvm, gfn, *pfn, (kvm_pfn_t )1 << *max_order); -#endif folio_unlock (folio);folio_put (folio);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 01/45] KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
Quoted 17 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Sean Christopherson <[email protected]>
Add and use
kvm_arch_has_gmem_convert()to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleedguest_memfddetails into arch code and would be far worse than adding yet anotherkvm_arch_has... hook.Opportunistically provide the
kvm_arch_gmem_make_private()declaration, and rely on dead-code elimination to eliminate the call to non-existent code whenCONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
Quoted 5 lines
Reported-by Binbin Wu <[email protected]>Suggested-by Ackerley Tng <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 59 lines
--- arch/x86/include/asm/kvm_host.h | 3 +++ include/linux/kvm_host.h | 3 ++- virt/kvm/guest_memfd.c | 5 ++--- 3 files changed, 7 insertions(+), 4 deletions(-)arch/x86/include/asm/kvm_host.h +3
-0@@ -1854,6 +1854,9 @@ enum kvm_intr_type {#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem (kvm) ((kvm)->arch.has_private_mem)#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert () (!!kvm_x86_ops.gmem_make_private) +#endif #define kvm_arch_has_readonly_mem (kvm) (!(kvm)->arch.has_protected_state)include/linux/kvm_host.h +2
-1@@ -2597,9 +2597,10 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, }#endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,kvm_pfn_t nr_pages); +#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert () false#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE virt/kvm/guest_memfd.c +2
-4@@ -772,11 +772,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT -if (kvm_gmem_is_private_mem (file_inode (file), index)) +if (kvm_arch_has_gmem_convert () && +kvm_gmem_is_private_mem (file_inode (file), index)) r =kvm_arch_gmem_make_private (kvm, gfn, *pfn, (kvm_pfn_t )1 << *max_order); -#endif folio_unlock (folio);folio_put (folio); --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 01/45] KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
In reply to Ackerley TngQuoted 16 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Add and use
kvm_arch_has_gmem_convert()to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleedguest_memfddetails into arch code and would be far worse than adding yet anotherkvm_arch_has... hook.Opportunistically provide the
kvm_arch_gmem_make_private()declaration, and rely on dead-code elimination to eliminate the call to non-existent code whenCONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
Quoted 5 lines
Reported-by Binbin Wu <[email protected]>Suggested-by Ackerley Tng <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Re: [PATCH v12 01/45] KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination
In reply to Ackerley TngQuoted 16 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Add and use
kvm_arch_has_gmem_convert()to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleedguest_memfddetails into arch code and would be far worse than adding yet anotherkvm_arch_has... hook.Opportunistically provide the
kvm_arch_gmem_make_private()declaration, and rely on dead-code elimination to eliminate the call to non-existent code whenCONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
Quoted 6 lines
Reported-by Binbin Wu <[email protected]>Suggested-by Ackerley Tng <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v12 2/45 KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Use kvm_mem_is_private() when populating guest_memfd instead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.
No functional change intended.
--- virt/kvm/guest_memfd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/guest_memfd.c +1 -3
@@ -805,9 +805,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,folio_unlock (folio); -if (!kvm_range_has_memory_attributes (kvm, gfn, gfn +1 , -KVM_MEMORY_ATTRIBUTE_PRIVATE , -KVM_MEMORY_ATTRIBUTE_PRIVATE )) { +if (!kvm_mem_is_private (kvm, gfn)) { ret = -EINVAL ; goto out_put_folio; }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 02/45] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
Quoted 9 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Use
kvm_mem_is_private()when populatingguest_memfdinstead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.No functional change intended.
Quoted 7 lines
Signed-off-by Sean Christopherson <[email protected]>Suggested-by Xiaoyao Li <[email protected]>Reviewed-by Xiaoyao Li <[email protected]>Reviewed-by Binbin Wu <[email protected]>Reviewed-by Fuad Tabba <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v12 3/45 KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Start plumbing in guest_memfd support for in-place private<=>shared conversions by tracking attributes via a maple tree. KVM currently tracks private vs. shared attributes on a per-VM basis, which made sense when a guest_memfd _only_ supported private memory, but tracking per-VM simply can't work for in-place conversions as the shared/private status of a given page needs to be per-gmem_inode, not per-VM.
Use the filemap invalidation lock to protect the maple tree, as taking the lock for read when faulting in memory (for userspace or the guest) isn't expected to result in meaningful contention, and using a separate lock would add significant complexity (avoiding deadlock is quite difficult).
In kvm_gmem_get_pfn(), drop the folio refcount before releasing filemap_invalidate_lock(). This ensures that a competing conversion request from userspace (to be added in a later patch), which also takes the filemap_invalidate_lock(), will never see an elevated refcount due to kvm_gmem_get_pfn().
--- virt/kvm/guest_memfd.c | 136 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 119 insertions(+), 17 deletions(-)
virt/kvm/guest_memfd.c +119 -17
@@ -4,6 +4,7 @@#include <linux/falloc.h> #include <linux/fs.h> #include <linux/kvm_host.h> +#include <linux/maple_tree.h> #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h>
@@ -34,6 +35,13 @@ struct gmem_inode {struct list_head gmem_file_list;u64 flags; +/* +* Every index in this inode, whether memory is populated or +* not, is tracked in attributes. The entire range of indices, +* corresponding to the size of this inode, is represented in +* this maple tree. +*/ +struct maple_tree attributes; }; static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
@@ -61,9 +69,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static u64 kvm_gmem_get_default_attributes (struct inode *inode) +{ +bool init_shared =GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED ; + +return init_shared ?0 :KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} + +static u64 kvm_gmem_get_attributes (struct inode *inode,void *entry) +{ +if (WARN_ON_ONCE (!entry)) +return kvm_gmem_get_default_attributes (inode); + +return xa_to_value (entry); +} +static bool kvm_gmem_is_private_mem (struct inode *inode,pgoff_t index) { -return !(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED ); +struct maple_tree *mt = &GMEM_I (inode)->attributes; +void *entry =mtree_load (mt, index); + +return kvm_gmem_get_attributes (inode, entry) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; }static bool kvm_gmem_is_shared_mem (struct inode *inode,pgoff_t index)
@@ -364,10 +391,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)if (((loff_t )vmf->pgoff <<PAGE_SHIFT ) >=i_size_read (inode))return VM_FAULT_SIGBUS ; -if (!kvm_gmem_is_shared_mem (inode, vmf->pgoff)) -return VM_FAULT_SIGBUS ; +filemap_invalidate_lock_shared (inode->i_mapping); +if (kvm_gmem_is_shared_mem (inode, vmf->pgoff)) + folio =kvm_gmem_get_folio (inode, vmf->pgoff); +else + folio =ERR_PTR (-EACCES ); +filemap_invalidate_unlock_shared (inode->i_mapping); - folio =kvm_gmem_get_folio (inode, vmf->pgoff);if (IS_ERR (folio)) {if (PTR_ERR (folio) == -EAGAIN )return VM_FAULT_RETRY ;
@@ -520,6 +550,51 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)return true; } +static int kvm_gmem_init_inode (struct inode *inode,loff_t size,u64 flags) +{ +struct gmem_inode *gi =GMEM_I (inode); +MA_STATE (mas, &gi->attributes,0 , (size >>PAGE_SHIFT ) -1 ); +u64 attrs; +int r; + + inode->i_op = &kvm_gmem_iops; + inode->i_mapping->a_ops = &kvm_gmem_aops; + inode->i_mode |=S_IFREG ; + inode->i_size = size; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); + +/* +* guest_memfd memory is neither migratable nor swappable: set +* inaccessible to gate off both. +*/ +mapping_set_inaccessible (inode->i_mapping); +WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); + + gi->flags = flags; + +mt_set_external_lock (&gi->attributes, + &inode->i_mapping->invalidate_lock); + +/* +* Store default attributes for the entire gmem instance. Ensuring every +* index is represented in the maple tree at all times simplifies the +* conversion and merging logic. +*/ + attrs =kvm_gmem_get_default_attributes (inode); + +/* +* Acquire the invalidation lock purely to make lockdep happy. The +* maple tree library expects all stores to be protected via the lock, +* and the library can't know when the tree is reachable only by the +* caller, as is the case here. +*/ +filemap_invalidate_lock (inode->i_mapping); + r =mas_store_gfp (&mas,xa_mk_value (attrs),GFP_KERNEL ); +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) {static const char *name ="[kvm-gmem]" ;
@@ -550,16 +625,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_fops; } - inode->i_op = &kvm_gmem_iops; - inode->i_mapping->a_ops = &kvm_gmem_aops; - inode->i_mode |=S_IFREG ; - inode->i_size = size; -mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); -mapping_set_inaccessible (inode->i_mapping); -/* Unmovable mappings are supposed to be marked unevictable as well. */ -WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); - -GMEM_I (inode)->flags = flags; + err =kvm_gmem_init_inode (inode, size, flags); +if (err) + goto err_inode; file =alloc_file_pseudo (inode, kvm_gmem_mnt, name,O_RDWR , &kvm_gmem_fops);if (IS_ERR (file)) {
@@ -763,9 +831,13 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,if (!file)return -EFAULT ; +filemap_invalidate_lock_shared (file_inode (file)->i_mapping); + folio =__kvm_gmem_get_pfn (file, slot, index, pfn, max_order); -if (IS_ERR (folio)) -return PTR_ERR (folio); +if (IS_ERR (folio)) { + r =PTR_ERR (folio); + goto out; + }if (!folio_test_uptodate (folio)) {clear_highpage (folio_page (folio,0 ));
@@ -780,6 +852,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_unlock (folio);folio_put (folio); +out: +filemap_invalidate_unlock_shared (file_inode (file)->i_mapping);return r; }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_gmem_get_pfn);
@@ -909,6 +983,15 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)mpol_shared_policy_init (&gi->policy,NULL ); +/* +* Memory attributes are protected by the filemap invalidation lock, but +* the lock structure isn't available at this time. Immediately mark +* maple tree as using external locking so that accessing the tree +* before it's fully initialized results in NULL pointer dereferences +* and not more subtle bugs. +*/ +mt_init_flags (&gi->attributes,MT_FLAGS_LOCK_EXTERN |MT_FLAGS_USE_RCU ); + gi->flags =0 ;INIT_LIST_HEAD (&gi->gmem_file_list);return &gi->vfs_inode;
@@ -916,7 +999,26 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)static void kvm_gmem_destroy_inode (struct inode *inode) { -mpol_free_shared_policy (&GMEM_I (inode)->policy); +struct gmem_inode *gi =GMEM_I (inode); + +mpol_free_shared_policy (&gi->policy); + +/* +* Note! Checking for an empty tree is functionally necessary +* to avoid explosions if the tree hasn't been fully +* initialized, i.e. if the inode is being destroyed before +* guest_memfd can set the external lock, lockdep would find +* that the tree's internal ma_lock was not held. +*/ +if (!mtree_empty (&gi->attributes)) { +/* +* Acquire the invalidation lock purely to make lockdep happy, +* the inode is unreachable at this point. +*/ +filemap_invalidate_lock (inode->i_mapping); +__mt_destroy (&gi->attributes); +filemap_invalidate_unlock (inode->i_mapping); + } }static void kvm_gmem_free_inode (struct inode *inode)
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 4/45 KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Rename the per-VM memory attributes Kconfig to make it explicitly about per-VM attributes in anticipation of adding memory attributes support to guest_memfd, at which point it will be possible (and desirable) to have memory attributes without the per-VM support, even in x86.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 6 +++--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/x86.c | 2 +- include/linux/kvm_host.h | 8 ++++---- include/trace/events/kvm.h | 4 ++-- virt/kvm/Kconfig | 2 +- virt/kvm/kvm_main.c | 14 +++++++------- 8 files changed, 20 insertions(+), 20 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1851,7 +1851,7 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem (kvm) ((kvm)->arch.has_private_mem)#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
arch/x86/kvm/Kconfig +3 -3
@@ -84,7 +84,7 @@ config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT depends on KVM_X86 && X86_64 - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES help Enable support for KVM software-protected VMs. Currently, software- protected VMs are purely a development and testing vehicle for
@@ -135,7 +135,7 @@ config KVM_INTEL_TDX bool "Intel Trust Domain Extensions (TDX) support" default y depends on INTEL_TDX_HOST - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_POPULATE help Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -159,7 +159,7 @@ config KVM_AMD_SEV depends on KVM_AMD && X86_64 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -8097,7 +8097,7 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)vhost_task_stop (kvm->arch.nx_huge_page_recovery_thread); } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static bool hugepage_test_mixed (struct kvm_memory_slot *slot,gfn_t gfn,int level) {
arch/x86/kvm/x86.c +1 -1
@@ -10140,7 +10140,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, } } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES kvm_mmu_init_memslot_memory_attributes (kvm, slot);#endif
include/linux/kvm_host.h +4 -4
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) }#endif -#ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static inline bool kvm_arch_has_private_mem (struct kvm *kvm) {return false;
@@ -872,7 +872,7 @@ struct kvm {#ifdef CONFIG_HAVE_KVM_PM_NOTIFIER struct notifier_block pm_notifier;#endif -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES /* Protected by slots_lock (for writes) and RCU (for reads) */ struct xarray mem_attr_array;#endif
@@ -2560,7 +2560,7 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot)return slot->flags &KVM_MEMSLOT_GMEM_ONLY ; } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static inline unsigned long kvm_get_memory_attributes (struct kvm *kvm,gfn_t gfn) {return xa_to_value (xa_load (&kvm->mem_attr_array, gfn));
@@ -2582,7 +2582,7 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) {return false; } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ #ifdef CONFIG_KVM_GUEST_MEMFD int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
include/trace/events/kvm.h +2 -2
@@ -358,7 +358,7 @@ TRACE_EVENT(kvm_dirty_ring_exit,TP_printk ("vcpu %d" , __entry->vcpu_id) ); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES /* * @start: Starting address of guest memory range * @end: End address of guest memory range
@@ -383,7 +383,7 @@ TRACE_EVENT(kvm_vm_set_mem_attributes,TP_printk ("%#016llx -- %#016llx [0x%lx]" , __entry->start, __entry->end, __entry->attr) ); -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ TRACE_EVENT(kvm_unmap_hva_range, TP_PROTO(unsigned long start, unsigned long end),
virt/kvm/Kconfig +1 -1
@@ -100,7 +100,7 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG config KVM_MMU_LOCKLESS_AGING bool -config KVM_GENERIC_MEMORY_ATTRIBUTES +config KVM_VM_MEMORY_ATTRIBUTES bool config KVM_GUEST_MEMFD
virt/kvm/kvm_main.c +7 -7
@@ -1116,7 +1116,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)spin_lock_init (&kvm->mn_invalidate_lock);rcuwait_init (&kvm->mn_memslots_update_rcuwait);xa_init (&kvm->vcpu_array); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES xa_init (&kvm->mem_attr_array);#endif
@@ -1301,7 +1301,7 @@ static void kvm_destroy_vm(struct kvm *kvm)cleanup_srcu_struct (&kvm->irq_srcu);srcu_barrier (&kvm->srcu);cleanup_srcu_struct (&kvm->srcu); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES xa_destroy (&kvm->mem_attr_array);#endif kvm_arch_free_vm (kvm);
@@ -2419,7 +2419,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, }#endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes (struct kvm *kvm) {if (!kvm ||kvm_arch_has_private_mem (kvm))
@@ -2624,7 +2624,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes); } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ struct kvm_memory_slot *gfn_to_memslot (struct kvm *kvm,gfn_t gfn) {
@@ -4941,7 +4941,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)case KVM_CAP_SYSTEM_EVENT_DATA :case KVM_CAP_DEVICE_CTRL :return 1 ; -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_CAP_MEMORY_ATTRIBUTES :return kvm_supported_mem_attributes (kvm);#endif
@@ -5345,7 +5345,7 @@ static long kvm_vm_ioctl(struct file *filp,break ; }#endif /* CONFIG_HAVE_KVM_IRQ_ROUTING */ -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_SET_MEMORY_ATTRIBUTES : {struct kvm_memory_attributes attrs;
@@ -5356,7 +5356,7 @@ static long kvm_vm_ioctl(struct file *filp, r =kvm_vm_ioctl_set_mem_attributes (kvm, &attrs); break; } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ case KVM_CREATE_DEVICE : {struct kvm_create_device cd;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 5/45 KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based on kvm_arch_has_private_mem being #defined in anticipation of tracking PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place conversion).
guest_memfd support for memory attributes is expected to be unconditional to avoid yet more macros (all architectures that support guest_memfd are expected to use per-gmem attributes at some point), at which point enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory attributes being supported by KVM at-large would result in a system-scope check (NULL @kvm) over-reporting support on arm64.
Give architectures full control over overriding the default definition of kvm_arch_has_private_mem() by removing the coupling with CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
In a later patch, kvm_arch_has_private_mem() will be defined based on whether architectural features are compiled in, and made orthogonal to CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
--- include/linux/kvm_host.h | 2 +- virt/kvm/kvm_main.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +1 -1
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) }#endif -#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +#ifndef kvm_arch_has_private_memstatic inline bool kvm_arch_has_private_mem (struct kvm *kvm) {return false;
virt/kvm/kvm_main.c +2 -0
@@ -2422,8 +2422,10 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes (struct kvm *kvm) { +#ifdef kvm_arch_has_private_memif (!kvm ||kvm_arch_has_private_mem (kvm))return KVM_MEMORY_ATTRIBUTE_PRIVATE ; +#endif return 0 ; }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 6/45 KVM: Rename memory attribute APIs to prepare for in-place gmem conversion
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Rename memory attribute APIs to add a "vm_" in the name in anticipation of moving PRIVATE tracking into guest_memfd, to allow in-place conversion between SHARED and PRIVATE. At that point, there will effectively be two (potential) sources of memory attributes: the VM and guest_memfd.
kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it is a helper local to the file.) Update the accompanying trace function to match.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 14 +++++++------- include/linux/kvm_host.h | 16 ++++++++-------- include/trace/events/kvm.h | 2 +- virt/kvm/kvm_main.c | 32 ++++++++++++++++---------------- 4 files changed, 32 insertions(+), 32 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -7
@@ -8116,8 +8116,8 @@ static void hugepage_set_mixed(struct kvm_memory_slot *slot, gfn_t gfn,lpage_info_slot (gfn, slot, level)->disallow_lpage |=KVM_LPAGE_MIXED_FLAG ; } -bool kvm_arch_pre_set_memory_attributes (struct kvm *kvm, -struct kvm_gfn_range *range) +bool kvm_arch_pre_set_vm_memory_attributes (struct kvm *kvm, +struct kvm_gfn_range *range) {struct kvm_memory_slot *slot = range->slot;int level;
@@ -8186,18 +8186,18 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,const unsigned long end = start +KVM_PAGES_PER_HPAGE (level);if (level ==PG_LEVEL_2M ) -return kvm_range_has_memory_attributes (kvm, start, end, ~0 , attrs); +return kvm_range_has_vm_memory_attributes (kvm, start, end, ~0 , attrs);for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 )) {if (hugepage_test_mixed (slot, gfn, level -1 ) || - attrs !=kvm_get_memory_attributes (kvm, gfn)) + attrs !=kvm_get_vm_memory_attributes (kvm, gfn))return false; }return true; } -bool kvm_arch_post_set_memory_attributes (struct kvm *kvm, -struct kvm_gfn_range *range) +bool kvm_arch_post_set_vm_memory_attributes (struct kvm *kvm, +struct kvm_gfn_range *range) {unsigned long attrs = range->arg.attributes;struct kvm_memory_slot *slot = range->slot;
@@ -8290,7 +8290,7 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,* be manually checked as the attributes may already be mixed. */ for (gfn = start; gfn < end; gfn += nr_pages) { -unsigned long attrs =kvm_get_memory_attributes (kvm, gfn); +unsigned long attrs =kvm_get_vm_memory_attributes (kvm, gfn);if (hugepage_has_attrs (kvm, slot, gfn, level, attrs))hugepage_clear_mixed (slot, gfn, level);
include/linux/kvm_host.h +8 -8
@@ -2561,21 +2561,21 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot) }#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static inline unsigned long kvm_get_memory_attributes (struct kvm *kvm,gfn_t gfn) +static inline unsigned long kvm_get_vm_memory_attributes (struct kvm *kvm,gfn_t gfn) {return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)); } -bool kvm_range_has_memory_attributes (struct kvm *kvm,gfn_t start,gfn_t end, -unsigned long mask,unsigned long attrs); -bool kvm_arch_pre_set_memory_attributes (struct kvm *kvm, -struct kvm_gfn_range *range); -bool kvm_arch_post_set_memory_attributes (struct kvm *kvm, -struct kvm_gfn_range *range); +bool kvm_range_has_vm_memory_attributes (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long mask,unsigned long attrs); +bool kvm_arch_pre_set_vm_memory_attributes (struct kvm *kvm, +struct kvm_gfn_range *range); +bool kvm_arch_post_set_vm_memory_attributes (struct kvm *kvm, +struct kvm_gfn_range *range);static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) { -return kvm_get_memory_attributes (kvm, gfn) &KVM_MEMORY_ATTRIBUTE_PRIVATE ; +return kvm_get_vm_memory_attributes (kvm, gfn) &KVM_MEMORY_ATTRIBUTE_PRIVATE ; }#else static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn)
include/trace/events/kvm.h +1 -1
@@ -364,7 +364,7 @@ TRACE_EVENT(kvm_dirty_ring_exit,* @end: End address of guest memory range * @attr: The value of the attribute being set. */ -TRACE_EVENT(kvm_vm_set_mem_attributes, +TRACE_EVENT(kvm_set_vm_mem_attributes, TP_PROTO(gfn_t start, gfn_t end, unsigned long attr), TP_ARGS(start, end, attr),
virt/kvm/kvm_main.c +16 -16
@@ -2420,7 +2420,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,#endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static u64 kvm_supported_mem_attributes (struct kvm *kvm) +static u64 kvm_supported_vm_mem_attributes (struct kvm *kvm) {#ifdef kvm_arch_has_private_memif (!kvm ||kvm_arch_has_private_mem (kvm))
@@ -2434,19 +2434,19 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)* Returns true if _all_ gfns in the range [@start, @end) have attributes * such that the bits in @mask match @attrs. */ -bool kvm_range_has_memory_attributes (struct kvm *kvm,gfn_t start,gfn_t end, -unsigned long mask,unsigned long attrs) +bool kvm_range_has_vm_memory_attributes (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long mask,unsigned long attrs) {XA_STATE (xas, &kvm->mem_attr_array, start);unsigned long index;void *entry; - mask &=kvm_supported_mem_attributes (kvm); + mask &=kvm_supported_vm_mem_attributes (kvm);if (attrs & ~mask)return false;if (end == start +1 ) -return (kvm_get_memory_attributes (kvm, start) & mask) == attrs; +return (kvm_get_vm_memory_attributes (kvm, start) & mask) == attrs;guard (rcu)();if (!attrs)
@@ -2517,8 +2517,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,KVM_MMU_UNLOCK (kvm); } -static bool kvm_pre_set_memory_attributes (struct kvm *kvm, -struct kvm_gfn_range *range) +static bool kvm_pre_set_vm_memory_attributes (struct kvm *kvm, +struct kvm_gfn_range *range) {/* * Unconditionally add the range to the invalidation set, regardless of
@@ -2533,18 +2533,18 @@ static bool kvm_pre_set_memory_attributes(struct kvm *kvm,*/ kvm_mmu_invalidate_range_add (kvm, range->start, range->end); -return kvm_arch_pre_set_memory_attributes (kvm, range); +return kvm_arch_pre_set_vm_memory_attributes (kvm, range); }/* Set @attributes for the gfn range [@start, @end). */ -static int kvm_vm_set_mem_attributes (struct kvm *kvm,gfn_t start,gfn_t end, +static int kvm_set_vm_mem_attributes (struct kvm *kvm,gfn_t start,gfn_t end,unsigned long attributes) {struct kvm_mmu_notifier_range pre_set_range = { .start = start, .end = end, .arg.attributes = attributes, - .handler = kvm_pre_set_memory_attributes, + .handler = kvm_pre_set_vm_memory_attributes, .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = true,
@@ -2553,7 +2553,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, .start = start, .end = end, .arg.attributes = attributes, - .handler = kvm_arch_post_set_memory_attributes, + .handler = kvm_arch_post_set_vm_memory_attributes, .on_lock = kvm_mmu_invalidate_end, .may_block = true, };
@@ -2563,12 +2563,12 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, entry = attributes ?xa_mk_value (attributes) :NULL ; -trace_kvm_vm_set_mem_attributes (start, end, attributes); +trace_kvm_set_vm_mem_attributes (start, end, attributes);mutex_lock (&kvm->slots_lock);/* Nothing to do if the entire range has the desired attributes. */ -if (kvm_range_has_memory_attributes (kvm, start, end, ~0 , attributes)) +if (kvm_range_has_vm_memory_attributes (kvm, start, end, ~0 , attributes)) goto out_unlock;/*
@@ -2607,7 +2607,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,/* flags is currently not used. */ if (attrs->flags)return -EINVAL ; -if (attrs->attributes & ~kvm_supported_mem_attributes (kvm)) +if (attrs->attributes & ~kvm_supported_vm_mem_attributes (kvm))return -EINVAL ;if (attrs->size ==0 || attrs->address + attrs->size < attrs->address)return -EINVAL ;
@@ -2624,7 +2624,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,*/ BUILD_BUG_ON (sizeof (attrs->attributes) !=sizeof (unsigned long )); - return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes); + return kvm_set_vm_mem_attributes(kvm, start, end, attrs->attributes); }#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
@@ -4945,7 +4945,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)return 1 ;#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_CAP_MEMORY_ATTRIBUTES : -return kvm_supported_mem_attributes (kvm); +return kvm_supported_vm_mem_attributes (kvm);#endif #ifdef CONFIG_KVM_GUEST_MEMFD case KVM_CAP_GUEST_MEMFD :
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 7/45 KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
In reply to Ackerley TngRename kvm_mem_is_private() to kvm_is_private_gfn() to prepare for in-place conversion, where there will be two lookup functions, kvm_vm_is_private_gfn() and kvm_gmem_is_private_gfn().
This renaming allows consistent prefixing of "vm" vs "gmem" for kvm_*_is_private_gfn(), as opposed to kvm_gmem_mem_is_private(), which looks like a typo.
--- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ arch/x86/kvm/svm/sev.c | 2 +- include/linux/kvm_host.h | 4 ++-- virt/kvm/guest_memfd.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +6 -6
@@ -3421,7 +3421,7 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_page_fault *fault, is_private = fault->is_private; }else { max_level =PG_LEVEL_NUM ; - is_private =kvm_mem_is_private (kvm, gfn); + is_private =kvm_is_private_gfn (kvm, gfn); } max_level =min (max_level, max_huge_page_level);
@@ -3634,13 +3634,13 @@ static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault* guest spinning on a #PF indefinitely, so don't attempt the fast path * in this case. * -* Note that the kvm_mem_is_private() check might race with an +* Note that the kvm_is_private_gfn() check might race with an * attribute update, but this will either result in the guest spinning * on RET_PF_SPURIOUS until the update completes, or an actual spurious * case might go down the slow path. Either case will resolve itself. */ if (kvm->arch.has_private_mem && - fault->is_private !=kvm_mem_is_private (kvm, fault->gfn)) + fault->is_private !=kvm_is_private_gfn (kvm, fault->gfn))return false;/*
@@ -4708,7 +4708,7 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,* Now that we have a snapshot of mmu_invalidate_seq we can check for a * private vs. shared mismatch. */ -if (fault->is_private !=kvm_mem_is_private (kvm, fault->gfn)) { +if (fault->is_private !=kvm_is_private_gfn (kvm, fault->gfn)) {kvm_mmu_prepare_memory_fault_exit (vcpu, fault);return -EFAULT ; }
@@ -5111,7 +5111,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, direct_bits =0 ;if (kvm_arch_has_private_mem (vcpu->kvm) && -kvm_mem_is_private (vcpu->kvm,gpa_to_gfn (range->gpa))) +kvm_is_private_gfn (vcpu->kvm,gpa_to_gfn (range->gpa))) error_code |=PFERR_PRIVATE_ACCESS ;else direct_bits =gfn_to_gpa (kvm_gfn_direct_bits (vcpu->kvm));
@@ -6584,7 +6584,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errif (IS_ENABLED (CONFIG_KVM_SW_PROTECTED_VM ) && !(error_code &PFERR_RSVD_MASK ) && vcpu->kvm->arch.vm_type ==KVM_X86_SW_PROTECTED_VM && -kvm_mem_is_private (vcpu->kvm,gpa_to_gfn (cr2_or_gpa))) +kvm_is_private_gfn (vcpu->kvm,gpa_to_gfn (cr2_or_gpa))) error_code |=PFERR_PRIVATE_ACCESS ; r =RET_PF_INVALID ;
arch/x86/kvm/svm/sev.c +1 -1
@@ -5030,7 +5030,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)* userspace via KVM_EXIT_MEMORY_FAULT events, however, so RMP faults * for shared pages should not end up here. */ -if (!kvm_mem_is_private (kvm, gfn)) { +if (!kvm_is_private_gfn (kvm, gfn)) {pr_warn_ratelimited ("SEV: Unexpected RMP fault for non-private GPA 0x%llx\n" , gpa);return ;
include/linux/kvm_host.h +2 -2
@@ -2573,12 +2573,12 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,bool kvm_arch_post_set_vm_memory_attributes (struct kvm *kvm,struct kvm_gfn_range *range); -static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return kvm_get_vm_memory_attributes (kvm, gfn) &KVM_MEMORY_ATTRIBUTE_PRIVATE ; }#else -static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false; }
virt/kvm/guest_memfd.c +1 -1
@@ -879,7 +879,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,folio_unlock (folio); -if (!kvm_mem_is_private (kvm, gfn)) { +if (!kvm_is_private_gfn (kvm, gfn)) { ret = -EINVAL ; goto out_put_folio; }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 07/45] KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
Quoted 7 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
Rename
kvm_mem_is_private()tokvm_is_private_gfn()to prepare for in-place conversion, where there will be two lookup functions,kvm_vm_is_private_gfn()andkvm_gmem_is_private_gfn().
Quoted 6 lines
This renaming allows consistent prefixing of "vm" vs "gmem" for kvm_*_is_private_gfn(), as opposed to
kvm_gmem_mem_is_private(), which looks like a typo.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 113 lines
--- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ arch/x86/kvm/svm/sev.c | 2 +- include/linux/kvm_host.h | 4 ++-- virt/kvm/guest_memfd.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)arch/x86/kvm/mmu/mmu.c +6
-6@@ -3421,7 +3421,7 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_page_fault *fault, is_private = fault->is_private; }else { max_level =PG_LEVEL_NUM ; - is_private =kvm_mem_is_private (kvm, gfn); + is_private =kvm_is_private_gfn (kvm, gfn); } max_level =min (max_level, max_huge_page_level);@@ -3634,13 +3634,13 @@ static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault* guest spinning on a #PF indefinitely, so don't attempt the fast path * in this case. * -* Note that the kvm_mem_is_private() check might race with an +* Note that the kvm_is_private_gfn() check might race with an * attribute update, but this will either result in the guest spinning * on RET_PF_SPURIOUS until the update completes, or an actual spurious * case might go down the slow path. Either case will resolve itself. */ if (kvm->arch.has_private_mem && - fault->is_private !=kvm_mem_is_private (kvm, fault->gfn)) + fault->is_private !=kvm_is_private_gfn (kvm, fault->gfn))return false;/* @@ -4708,7 +4708,7 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,* Now that we have a snapshot of mmu_invalidate_seq we can check for a * private vs. shared mismatch. */ -if (fault->is_private !=kvm_mem_is_private (kvm, fault->gfn)) { +if (fault->is_private !=kvm_is_private_gfn (kvm, fault->gfn)) {kvm_mmu_prepare_memory_fault_exit (vcpu, fault);return -EFAULT ; }@@ -5111,7 +5111,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, direct_bits =0 ;if (kvm_arch_has_private_mem (vcpu->kvm) && -kvm_mem_is_private (vcpu->kvm,gpa_to_gfn (range->gpa))) +kvm_is_private_gfn (vcpu->kvm,gpa_to_gfn (range->gpa))) error_code |=PFERR_PRIVATE_ACCESS ;else direct_bits =gfn_to_gpa (kvm_gfn_direct_bits (vcpu->kvm));@@ -6584,7 +6584,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errif (IS_ENABLED (CONFIG_KVM_SW_PROTECTED_VM ) && !(error_code &PFERR_RSVD_MASK ) && vcpu->kvm->arch.vm_type ==KVM_X86_SW_PROTECTED_VM && -kvm_mem_is_private (vcpu->kvm,gpa_to_gfn (cr2_or_gpa))) +kvm_is_private_gfn (vcpu->kvm,gpa_to_gfn (cr2_or_gpa))) error_code |=PFERR_PRIVATE_ACCESS ; r =RET_PF_INVALID ;arch/x86/kvm/svm/sev.c +1
-1@@ -5030,7 +5030,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)* userspace via KVM_EXIT_MEMORY_FAULT events, however, so RMP faults * for shared pages should not end up here. */ -if (!kvm_mem_is_private (kvm, gfn)) { +if (!kvm_is_private_gfn (kvm, gfn)) {pr_warn_ratelimited ("SEV: Unexpected RMP fault for non-private GPA 0x%llx\n" , gpa);return ;include/linux/kvm_host.h +2
-2@@ -2573,12 +2573,12 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,bool kvm_arch_post_set_vm_memory_attributes (struct kvm *kvm,struct kvm_gfn_range *range); -static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return kvm_get_vm_memory_attributes (kvm, gfn) &KVM_MEMORY_ATTRIBUTE_PRIVATE ; }#else -static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false; }virt/kvm/guest_memfd.c +1
-2@@ -879,7 +879,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,folio_unlock (folio); -if (!kvm_mem_is_private (kvm, gfn)) { +if (!kvm_is_private_gfn (kvm, gfn)) { ret = -EINVAL ; goto out_put_folio; } --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 07/45] KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn()
In reply to Ackerley TngQuoted 6 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
Rename
kvm_mem_is_private()tokvm_is_private_gfn()to prepare for in-place conversion, where there will be two lookup functions,kvm_vm_is_private_gfn()andkvm_gmem_is_private_gfn().
Quoted 6 lines
This renaming allows consistent prefixing of "vm" vs "gmem" for kvm_*_is_private_gfn(), as opposed to
kvm_gmem_mem_is_private(), which looks like a typo.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
PATCH v12 8/45 KVM: Provide generic interface for checking memory private/shared status
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Introduce a generic kvm_is_private_gfn() interface using a static call to determine if a GFN is private. This allows the implementation for checking a GFN's private/shared status to be set at runtime.
In preparation for choosing implementations between a guest_memfd lookup and the existing VM attribute lookup, rename the existing VM-attribute-based check to kvm_vm_is_private_gfn() to emphasize that it looks up VM attributes.
--- include/linux/kvm_host.h | 14 ++++++++++++-- virt/kvm/kvm_main.c | 15 +++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-)
include/linux/kvm_host.h +12 -2
@@ -2573,16 +2573,26 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,bool kvm_arch_post_set_vm_memory_attributes (struct kvm *kvm,struct kvm_gfn_range *range); -static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) +static inline bool kvm_vm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return kvm_get_vm_memory_attributes (kvm, gfn) &KVM_MEMORY_ATTRIBUTE_PRIVATE ; } +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ + +#ifdef kvm_arch_has_private_mem +typedef bool (kvm_is_private_gfn_t )(struct kvm *kvm,gfn_t gfn); +DECLARE_STATIC_CALL (__kvm_is_private_gfn, kvm_is_private_gfn_t); + +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) +{ +return static_call (__kvm_is_private_gfn)(kvm, gfn); +}#else static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false; } -#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ +#endif /* kvm_arch_has_private_mem */ #ifdef CONFIG_KVM_GUEST_MEMFD int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
virt/kvm/kvm_main.c +15 -0
@@ -2628,6 +2628,20 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, }#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ +#ifdef kvm_arch_has_private_mem +DEFINE_STATIC_CALL_RET0 (__kvm_is_private_gfn, kvm_is_private_gfn_t); +EXPORT_STATIC_CALL_GPL (__kvm_is_private_gfn); + +static void kvm_init_memory_attributes (void ) +{ +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +static_call_update (__kvm_is_private_gfn, kvm_vm_is_private_gfn); +#endif +} +#else +static void kvm_init_memory_attributes (void ) { } +#endif +struct kvm_memory_slot *gfn_to_memslot (struct kvm *kvm,gfn_t gfn) {return __gfn_to_memslot (kvm_memslots (kvm), gfn);
@@ -6541,6 +6555,7 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) kvm_preempt_ops.sched_in = kvm_sched_in; kvm_preempt_ops.sched_out = kvm_sched_out; +kvm_init_memory_attributes ();kvm_init_debug (); r =kvm_vfio_ops_init ();
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 9/45 KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_is_private_gfn() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.
--- Documentation/virt/kvm/api.rst | 7 +++++++ include/linux/kvm_host.h | 6 ++++++ virt/kvm/guest_memfd.c | 26 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 12 +++++++++++- 4 files changed, 50 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +7 -0
@@ -6382,6 +6382,11 @@ KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes +userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the +sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` +to find out more about the creation-time shared/private status. + +If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE state. At VM creation time, all memory is shared, i.e. the PRIVATE attribute is '0' for all gfns. Userspace can control whether memory is shared/private by
@@ -6429,6 +6434,8 @@ the state of a gfn/page as needed. The "flags" field is reserved for future extensions and must be '0'. +.. _KVM_CREATE_GUEST_MEMFD: + 4.142 KVM_CREATE_GUEST_MEMFD ----------------------------
include/linux/kvm_host.h +6 -0
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ #ifdef kvm_arch_has_private_mem +extern bool gmem_in_place_conversion; +typedef bool (kvm_is_private_gfn_t )(struct kvm *kvm,gfn_t gfn);DECLARE_STATIC_CALL (__kvm_is_private_gfn, kvm_is_private_gfn_t);
@@ -2588,6 +2590,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) return static_call(__kvm_is_private_gfn)(kvm, gfn); }#else +#define gmem_in_place_conversion false +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#endif /* kvm_arch_has_private_mem */ #ifdef CONFIG_KVM_GUEST_MEMFD +bool kvm_gmem_is_private_gfn (struct kvm *kvm,gfn_t gfn); +int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,int *max_order);#else
virt/kvm/guest_memfd.c +26 -0
@@ -478,6 +478,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)return 0 ; } +bool kvm_gmem_is_private_gfn (struct kvm *kvm,gfn_t gfn) +{ +struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); +struct inode *inode; + +if (!slot) +return 0 ; + +CLASS (gmem_get_file, file)(slot); +if (!file) +return 0 ; + + inode =file_inode (file); + +/* +* Rely on the maple tree's internal RCU lock to ensure a stable result. +* This result can become stale as soon as the lock is dropped, so the +* caller _must_ protect consumption of private vs. shared either by +* holding guest_memfd's invalidate lock for the entire duration, or by +* checking mmu_invalidate_retry_gfn() under mmu_lock to serialize +* against concurrent attribute updates. +*/ +return kvm_gmem_is_private_mem (inode,kvm_gmem_get_index (slot, gfn)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_gmem_is_private_gfn); +static struct file_operations kvm_gmem_fops = { .mmap = kvm_gmem_mmap, .open = generic_file_open,
virt/kvm/kvm_main.c +11 -1
@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 ); +#ifdef kvm_arch_has_private_mem +bool __ro_after_init gmem_in_place_conversion = false; +#endif +/* * Ordering of locks: *
@@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,static u64 kvm_supported_vm_mem_attributes (struct kvm *kvm) {#ifdef kvm_arch_has_private_mem +if (gmem_in_place_conversion) +return 0 ; +if (!kvm ||kvm_arch_has_private_mem (kvm))return KVM_MEMORY_ATTRIBUTE_PRIVATE ;#endif
@@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_is_private_gfn);static void kvm_init_memory_attributes (void ) { +if (gmem_in_place_conversion) +static_call_update (__kvm_is_private_gfn, kvm_gmem_is_private_gfn);#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static_call_update (__kvm_is_private_gfn, kvm_vm_is_private_gfn); + else +static_call_update (__kvm_is_private_gfn ,kvm_vm_is_private_gfn );#endif }#else
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 09/45] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
Quoted 9 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Sean Christopherson <[email protected]>
Stub in global variable to enable in-place
guest_memfdprivate<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the__kvm_is_private_gfn()static call to theguest_memfdversion when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.
Quoted 5 lines
Cc Fuad Tabba <[email protected]>Cc Xiaoyao Li <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Ackerley Tng <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 143 lines
--- Documentation/virt/kvm/api.rst | 7 +++++++ include/linux/kvm_host.h | 6 ++++++ virt/kvm/guest_memfd.c | 26 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 12 +++++++++++- 4 files changed, 50 insertions(+), 1 deletion(-)Documentation/virt/kvm/api.rst +7
-0@@ -6382,6 +6382,11 @@ KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes +userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the +sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` +to find out more about the creation-time shared/private status. + +If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE state. At VM creation time, all memory is shared, i.e. the PRIVATE attribute is '0' for all gfns. Userspace can control whether memory is shared/private by@@ -6429,6 +6434,8 @@ the state of a gfn/page as needed. The "flags" field is reserved for future extensions and must be '0'. +.. _KVM_CREATE_GUEST_MEMFD: + 4.142 KVM_CREATE_GUEST_MEMFD ----------------------------include/linux/kvm_host.h +6
-0@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ #ifdef kvm_arch_has_private_mem +extern bool gmem_in_place_conversion; +typedef bool (kvm_is_private_gfn_t )(struct kvm *kvm,gfn_t gfn);DECLARE_STATIC_CALL (__kvm_is_private_gfn, kvm_is_private_gfn_t);@@ -2588,6 +2590,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) return static_call(__kvm_is_private_gfn)(kvm, gfn); }#else +#define gmem_in_place_conversion false +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false;@@ -2595,6 +2599,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#endif /* kvm_arch_has_private_mem */ #ifdef CONFIG_KVM_GUEST_MEMFD +bool kvm_gmem_is_private_gfn (struct kvm *kvm,gfn_t gfn); +int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,int *max_order);#else virt/kvm/guest_memfd.c +26
-0@@ -478,6 +478,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)return 0 ; } +bool kvm_gmem_is_private_gfn (struct kvm *kvm,gfn_t gfn) +{ +struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); +struct inode *inode; + +if (!slot) +return 0 ; + +CLASS (gmem_get_file, file)(slot); +if (!file) +return 0 ; + + inode =file_inode (file); + +/* +* Rely on the maple tree's internal RCU lock to ensure a stable result. +* This result can become stale as soon as the lock is dropped, so the +* caller _must_ protect consumption of private vs. shared either by +* holding guest_memfd's invalidate lock for the entire duration, or by +* checking mmu_invalidate_retry_gfn() under mmu_lock to serialize +* against concurrent attribute updates. +*/ +return kvm_gmem_is_private_mem (inode,kvm_gmem_get_index (slot, gfn)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_gmem_is_private_gfn); +static struct file_operations kvm_gmem_fops = { .mmap = kvm_gmem_mmap, .open = generic_file_open,virt/kvm/kvm_main.c +11
-2@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 ); +#ifdef kvm_arch_has_private_mem +bool __ro_after_init gmem_in_place_conversion = false; +#endif +/* * Ordering of locks: * @@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,static u64 kvm_supported_vm_mem_attributes (struct kvm *kvm) {#ifdef kvm_arch_has_private_mem +if (gmem_in_place_conversion) +return 0 ; +if (!kvm ||kvm_arch_has_private_mem (kvm))return KVM_MEMORY_ATTRIBUTE_PRIVATE ;#endif @@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_is_private_gfn);static void kvm_init_memory_attributes (void ) { +if (gmem_in_place_conversion) +static_call_update (__kvm_is_private_gfn, kvm_gmem_is_private_gfn);#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static_call_update (__kvm_is_private_gfn, kvm_vm_is_private_gfn); + else +static_call_update (__kvm_is_private_gfn ,kvm_vm_is_private_gfn );#endif }#else --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 09/45] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
In reply to Ackerley TngQuoted 8 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Stub in global variable to enable in-place
guest_memfdprivate<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the__kvm_is_private_gfn()static call to theguest_memfdversion when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.
Quoted 5 lines
Cc Fuad Tabba <[email protected]>Cc Xiaoyao Li <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Ackerley Tng <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
One nit below.
Quoted 12 lines
--- Documentation/virt/kvm/api.rst | 7 +++++++ include/linux/kvm_host.h | 6 ++++++ virt/kvm/guest_memfd.c | 26 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 12 +++++++++++- 4 files changed, 50 insertions(+), 1 deletion(-)Documentation/virt/kvm/api.rst +0
-0
Documentation/virt/kvm/api.rst KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated · line 6389
Quoted 8 lines
Documentation/virt/kvm/api.rst +5
-0@@ -6382,6 +6382,11 @@ KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes +userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the +sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` +to find out more about the creation-time shared/private status. + +If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes
Nit: i.e -> i.e.
Quoted 5 lines
Documentation/virt/kvm/api.rst +0
-0userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE state. At VM creation time, all memory is shared, i.e. the PRIVATE attribute is '0' for all gfns. Userspace can control whether memory is shared/private by@@ -6429,6 +6434,8 @@ the state of a gfn/page as needed.
Re: [PATCH v12 09/45] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
In reply to Ackerley TngQuoted 8 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Stub in global variable to enable in-place
guest_memfdprivate<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the__kvm_is_private_gfn()static call to theguest_memfdversion when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.
Quoted 6 lines
Cc Fuad Tabba <[email protected]>Cc Xiaoyao Li <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Ackerley Tng <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v12 10/45 KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Move the kvm_arch_has_private_mem() stub and a few guest_memfd function definitions/declarations "down" in kvm_host.h to utilize existing #ifdefs, and so that related code is clustered together.
No functional change intended.
--- include/linux/kvm_host.h | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-)
include/linux/kvm_host.h +16 -21
@@ -722,27 +722,6 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) }#endif -#ifndef kvm_arch_has_private_mem -static inline bool kvm_arch_has_private_mem (struct kvm *kvm) -{ -return false; -} -#endif - -#ifdef CONFIG_KVM_GUEST_MEMFD -bool kvm_arch_supports_gmem_init_shared (struct kvm *kvm); - -static inline u64 kvm_gmem_get_supported_flags (struct kvm *kvm) -{ -u64 flags =GUEST_MEMFD_FLAG_MMAP ; - -if (!kvm ||kvm_arch_supports_gmem_init_shared (kvm)) - flags |=GUEST_MEMFD_FLAG_INIT_SHARED ; - -return flags; -} -#endif -#ifndef kvm_arch_has_readonly_memstatic inline bool kvm_arch_has_readonly_mem (struct kvm *kvm) {
@@ -2592,6 +2571,11 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#else #define gmem_in_place_conversion false +static inline bool kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static inline bool kvm_is_private_gfn (struct kvm *kvm,gfn_t gfn) {return false;
@@ -2600,6 +2584,17 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)#ifdef CONFIG_KVM_GUEST_MEMFD bool kvm_gmem_is_private_gfn (struct kvm *kvm,gfn_t gfn); +bool kvm_arch_supports_gmem_init_shared (struct kvm *kvm); + +static inline u64 kvm_gmem_get_supported_flags (struct kvm *kvm) +{ +u64 flags =GUEST_MEMFD_FLAG_MMAP ; + +if (!kvm ||kvm_arch_supports_gmem_init_shared (kvm)) + flags |=GUEST_MEMFD_FLAG_INIT_SHARED ; + +return flags; +}int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,int *max_order);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 11/45 KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
When removing one or more folios from a guest_memfd instance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.
Opportunistically rename the helper to capture that it returns a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).
Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd).
The incorrect over-zapping of SHARED memory that doesn't belong to the gmem fd requesting the zapping will be resolved in a later patch, where, if in-place conversion is enabled, KVM will use both shared and private memory from the guest_memfd. If both shared and private memory are from the guest_memfd, invalidation will only zap memory belonging to the given gmem instance.
--- virt/kvm/guest_memfd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
virt/kvm/guest_memfd.c +6 -5
@@ -139,8 +139,11 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)return folio; } -static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter (struct inode *inode) +static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter (struct inode *inode) { +if (gmem_in_place_conversion) +return KVM_FILTER_SHARED |KVM_FILTER_PRIVATE ; +if (GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )return KVM_FILTER_SHARED ;
@@ -191,11 +194,9 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,static void kvm_gmem_invalidate_start (struct inode *inode,pgoff_t start,pgoff_t end) { -enum kvm_gfn_range_filter attr_filter; +enum kvm_gfn_range_filter attr_filter =kvm_gmem_get_all_gfns_filter (inode);struct gmem_file *f; - attr_filter =kvm_gmem_get_invalidate_filter (inode); -kvm_gmem_for_each_file (f, inode)__kvm_gmem_invalidate_start (f, start, end, attr_filter); }
@@ -347,7 +348,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)* memory, as its lifetime is associated with the inode, not the file. */ __kvm_gmem_invalidate_start (f,0 ,-1ul , -kvm_gmem_get_invalidate_filter (inode)); +kvm_gmem_get_all_gfns_filter (inode));__kvm_gmem_invalidate_end (f,0 ,-1ul );list_del (&f->entry);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 11/45] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
Quoted 28 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Sean Christopherson <[email protected]>
When removing one or more folios from a
guest_memfdinstance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.Opportunistically rename the helper to capture that it returns a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).
Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a
PUNCH_HOLEoperation on a PRIVATEguest_memfdwill incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd).The incorrect over-zapping of SHARED memory that doesn't belong to the gmem fd requesting the zapping will be resolved in a later patch, where, if in-place conversion is enabled, KVM will use both shared and private memory from the
guest_memfd. If both shared and private memory are from theguest_memfd, invalidation will only zap memory belonging to the given gmem instance.
Quoted 2 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 48 lines
--- virt/kvm/guest_memfd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)virt/kvm/guest_memfd.c +6
-6@@ -139,8 +139,11 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)return folio; } -static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter (struct inode *inode) +static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter (struct inode *inode) { +if (gmem_in_place_conversion) +return KVM_FILTER_SHARED |KVM_FILTER_PRIVATE ; +if (GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )return KVM_FILTER_SHARED ;@@ -191,11 +194,9 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,static void kvm_gmem_invalidate_start (struct inode *inode,pgoff_t start,pgoff_t end) { -enum kvm_gfn_range_filter attr_filter; +enum kvm_gfn_range_filter attr_filter =kvm_gmem_get_all_gfns_filter (inode);struct gmem_file *f; - attr_filter =kvm_gmem_get_invalidate_filter (inode); -kvm_gmem_for_each_file (f, inode)__kvm_gmem_invalidate_start (f, start, end, attr_filter); }@@ -347,7 +348,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)* memory, as its lifetime is associated with the inode, not the file. */ __kvm_gmem_invalidate_start (f,0 ,-1ul , -kvm_gmem_get_invalidate_filter (inode)); +kvm_gmem_get_all_gfns_filter (inode));__kvm_gmem_invalidate_end (f,0 ,-1ul );list_del (&f->entry); --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 11/45] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
In reply to Ackerley TngQuoted 27 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
When removing one or more folios from a
guest_memfdinstance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.Opportunistically rename the helper to capture that it returns a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).
Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a
PUNCH_HOLEoperation on a PRIVATEguest_memfdwill incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd).The incorrect over-zapping of SHARED memory that doesn't belong to the gmem fd requesting the zapping will be resolved in a later patch, where, if in-place conversion is enabled, KVM will use both shared and private memory from the
guest_memfd. If both shared and private memory are from theguest_memfd, invalidation will only zap memory belonging to the given gmem instance.
Quoted 2 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Re: [PATCH v12 11/45] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions
In reply to Ackerley TngQuoted 27 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
When removing one or more folios from a
guest_memfdinstance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.Opportunistically rename the helper to capture that it returns a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).
Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a
PUNCH_HOLEoperation on a PRIVATEguest_memfdwill incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd).The incorrect over-zapping of SHARED memory that doesn't belong to the gmem fd requesting the zapping will be resolved in a later patch, where, if in-place conversion is enabled, KVM will use both shared and private memory from the
guest_memfd. If both shared and private memory are from theguest_memfd, invalidation will only zap memory belonging to the given gmem instance.
Quoted 3 lines
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v12 12/45 KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
In reply to Ackerley TngIf a guest_memfd memslot is created but the guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP set, KVM still fulfils guest faults by looking up the memslot's userspace_addr.
Set KVM_MEMSLOT_GMEM_ONLY if in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion, guest_memfd will be the only backing memory for the memslot.
No validation is performed to require userspace_addr to be a mapping from the associated guest_memfd because even after validation, userspace is free to remap something else at the provided userspace_addr.
userspace_addr will still be used by functions like kvm_read_guest(), and if userspace_addr does not match up with the corresponding memory in the memslot's guest_memfd (whether userspace_addr points to the wrong offset or some non-guest_memfd memory, etc), that is a user error.
Requiring both shared and private memory to come from the only associated guest_memfd simplifies invalidation in stage 2 page tables. On a PUNCH_HOLE operation on a guest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the given guest_memfd.
--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++-------- virt/kvm/guest_memfd.c | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-)
Documentation/virt/kvm/api.rst +14 -8
@@ -6381,10 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes -userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the -sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` -to find out more about the creation-time shared/private status. +When mapping a gfn into the guest, guest faults are always serviced from +guest_memfd regardless of whether memory is shared or private. KVM determines +shared vs. private based on the state in guest_memfd, which is the sole +authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to +find out more about the creation-time shared/private status. + +userspace_addr is expected to be the mmap()-ed address corresponding to the +right offset within the guest_memfd. Any mismatch between userspace_addr and +guest_memfd is not validated and is a user error. userspace_addr is only used +for host-side guest accesses such as kvm_read_guest(). If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
@@ -6490,10 +6496,10 @@ specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags: page tables. Private memory cannot. ============================ ================================================ -When the KVM MMU performs a PFN lookup to service a guest fault and the backing -guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be -consumed from guest_memfd, regardless of whether it is a shared or a private -fault. +When the KVM MMU performs a PFN lookup to service a guest fault, the fault will +always be consumed from guest_memfd, regardless of whether it is a shared or a +private fault (unless in-place conversion is disabled and the backing +guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP flag set). See KVM_SET_USER_MEMORY_REGION2 for additional details.
virt/kvm/guest_memfd.c +1 -1
@@ -746,7 +746,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,*/ WRITE_ONCE (slot->gmem.file, file); slot->gmem.pgoff = start; -if (kvm_gmem_supports_mmap (inode)) +if (gmem_in_place_conversion ||kvm_gmem_supports_mmap (inode)) slot->flags |=KVM_MEMSLOT_GMEM_ONLY ;xa_store_range (&f->bindings, start, end -1 , slot,GFP_KERNEL );
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 12/45] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
Quoted 21 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
If a
guest_memfdmemslot is created but theguest_memfddoes not have theGUEST_MEMFD_FLAG_MMAPset, KVM still fulfils guest faults by looking up the memslot'suserspace_addr.Set
KVM_MEMSLOT_GMEM_ONLYif in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion,guest_memfdwill be the only backing memory for the memslot.No validation is performed to require
userspace_addrto be a mapping from the associatedguest_memfdbecause even after validation, userspace is free to remap something else at the provideduserspace_addr.
userspace_addrwill still be used by functions likekvm_read_guest(), and ifuserspace_addrdoes not match up with the corresponding memory in the memslot'sguest_memfd(whetheruserspace_addrpoints to the wrong offset or some non-guest_memfd memory, etc), that is a user error.
Quoted 7 lines
Requiring both shared and private memory to come from the only associated
guest_memfdsimplifies invalidation in stage 2 page tables. On aPUNCH_HOLEoperation on aguest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the givenguest_memfd.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 63 lines
--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++-------- virt/kvm/guest_memfd.c | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-)Documentation/virt/kvm/api.rst +14
-8@@ -6381,10 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes -userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the -sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` -to find out more about the creation-time shared/private status. +When mapping a gfn into the guest, guest faults are always serviced from +guest_memfd regardless of whether memory is shared or private. KVM determines +shared vs. private based on the state in guest_memfd, which is the sole +authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to +find out more about the creation-time shared/private status. + +userspace_addr is expected to be the mmap()-ed address corresponding to the +right offset within the guest_memfd. Any mismatch between userspace_addr and +guest_memfd is not validated and is a user error. userspace_addr is only used +for host-side guest accesses such as kvm_read_guest(). If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE@@ -6490,10 +6496,10 @@ specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags: page tables. Private memory cannot. ============================ ================================================ -When the KVM MMU performs a PFN lookup to service a guest fault and the backing -guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be -consumed from guest_memfd, regardless of whether it is a shared or a private -fault. +When the KVM MMU performs a PFN lookup to service a guest fault, the fault will +always be consumed from guest_memfd, regardless of whether it is a shared or a +private fault (unless in-place conversion is disabled and the backing +guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP flag set). See KVM_SET_USER_MEMORY_REGION2 for additional details.virt/kvm/guest_memfd.c +1
-2@@ -746,7 +746,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,*/ WRITE_ONCE (slot->gmem.file, file); slot->gmem.pgoff = start; -if (kvm_gmem_supports_mmap (inode)) +if (gmem_in_place_conversion ||kvm_gmem_supports_mmap (inode)) slot->flags |=KVM_MEMSLOT_GMEM_ONLY ;xa_store_range (&f->bindings, start, end -1 , slot,GFP_KERNEL ); --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 12/45] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
In reply to Ackerley TngQuoted 77 lines · On Sun, Aug 30, 2026 at 05:25:13PM -0700, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
If a
guest_memfdmemslot is created but theguest_memfddoes not have theGUEST_MEMFD_FLAG_MMAPset, KVM still fulfils guest faults by looking up the memslot'suserspace_addr.Set
KVM_MEMSLOT_GMEM_ONLYif in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion,guest_memfdwill be the only backing memory for the memslot.No validation is performed to require
userspace_addrto be a mapping from the associatedguest_memfdbecause even after validation, userspace is free to remap something else at the provideduserspace_addr.
userspace_addrwill still be used by functions likekvm_read_guest(), and ifuserspace_addrdoes not match up with the corresponding memory in the memslot'sguest_memfd(whetheruserspace_addrpoints to the wrong offset or some non-guest_memfd memory, etc), that is a user error.Requiring both shared and private memory to come from the only associated
guest_memfdsimplifies invalidation in stage 2 page tables. On aPUNCH_HOLEoperation on aguest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the givenguest_memfd.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++-------- virt/kvm/guest_memfd.c | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-)Documentation/virt/kvm/api.rst +14
-8@@ -6381,10 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes -userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the -sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` -to find out more about the creation-time shared/private status. +When mapping a gfn into the guest, guest faults are always serviced from +guest_memfd regardless of whether memory is shared or private. KVM determines +shared vs. private based on the state in guest_memfd, which is the sole +authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to +find out more about the creation-time shared/private status. + +userspace_addr is expected to be the mmap()-ed address corresponding to the +right offset within the guest_memfd. Any mismatch between userspace_addr and +guest_memfd is not validated and is a user error. userspace_addr is only used +for host-side guest accesses such as kvm_read_guest(). If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE@@ -6490,10 +6496,10 @@ specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags: page tables. Private memory cannot. ============================ ================================================ -When the KVM MMU performs a PFN lookup to service a guest fault and the backing -guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be -consumed from guest_memfd, regardless of whether it is a shared or a private -fault. +When the KVM MMU performs a PFN lookup to service a guest fault, the fault will +always be consumed from guest_memfd, regardless of whether it is a shared or a +private fault (unless in-place conversion is disabled and the backing +guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP flag set). See KVM_SET_USER_MEMORY_REGION2 for additional details.virt/kvm/guest_memfd.c +0
-0
virt/kvm/guest_memfd.c int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · line 750
Quoted 6 lines
virt/kvm/guest_memfd.c +1
-1@@ -746,7 +746,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,*/ WRITE_ONCE (slot->gmem.file, file); slot->gmem.pgoff = start; -if (kvm_gmem_supports_mmap (inode)) +if (gmem_in_place_conversion ||kvm_gmem_supports_mmap (inode)) slot->flags |=KVM_MEMSLOT_GMEM_ONLY ;
I like this change, which actually enforces in-place conversion -- when gmem_in_place_conversion is true, if userspace sets slot->userspace_addr to a different backend, the host and guest will no longer be able to access the same backend.
However, since gmem_in_place_conversion is globally and statically specified, it essentially disables the coexistence of VMs using out-of-place conversions when gmem_in_place_conversion is true. Previously, VMs using out-of-place conversions could still boot successfully as long as they switched to using per-gmem memory attributes. Is this change intended?
On a separate note, is kvm_gmem_supports_mmap() still a necessary requirement for setting KVM_MEMSLOT_GMEM_ONLY? When gmem_in_place_conversion is false, if userspace mmap()s the gmem but sets slot->userspace_addr to a different backend, should the KVM_MEMSLOT_GMEM_ONLY flag still be set?
Quoted 7 lines
virt/kvm/guest_memfd.c +0
-0xa_store_range (&f->bindings, start, end -1 , slot,GFP_KERNEL );Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 12/45] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
Quoted 89 lines · Yan Zhao
Quoted 77 lines · On Sun, Aug 30, 2026 at 05:25:13PM -0700, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
If a
guest_memfdmemslot is created but theguest_memfddoes not have theGUEST_MEMFD_FLAG_MMAPset, KVM still fulfils guest faults by looking up the memslot'suserspace_addr.Set
KVM_MEMSLOT_GMEM_ONLYif in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion,guest_memfdwill be the only backing memory for the memslot.No validation is performed to require
userspace_addrto be a mapping from the associatedguest_memfdbecause even after validation, userspace is free to remap something else at the provideduserspace_addr.
userspace_addrwill still be used by functions likekvm_read_guest(), and ifuserspace_addrdoes not match up with the corresponding memory in the memslot'sguest_memfd(whetheruserspace_addrpoints to the wrong offset or some non-guest_memfd memory, etc), that is a user error.Requiring both shared and private memory to come from the only associated
guest_memfdsimplifies invalidation in stage 2 page tables. On aPUNCH_HOLEoperation on aguest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the givenguest_memfd.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++-------- virt/kvm/guest_memfd.c | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-)Documentation/virt/kvm/api.rst +14
-8@@ -6381,10 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes -userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the -sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` -to find out more about the creation-time shared/private status. +When mapping a gfn into the guest, guest faults are always serviced from +guest_memfd regardless of whether memory is shared or private. KVM determines +shared vs. private based on the state in guest_memfd, which is the sole +authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to +find out more about the creation-time shared/private status. + +userspace_addr is expected to be the mmap()-ed address corresponding to the +right offset within the guest_memfd. Any mismatch between userspace_addr and +guest_memfd is not validated and is a user error. userspace_addr is only used +for host-side guest accesses such as kvm_read_guest(). If in-place conversion is disabled, KVM selects shared vs. private, i.e consumes userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE@@ -6490,10 +6496,10 @@ specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags: page tables. Private memory cannot. ============================ ================================================ -When the KVM MMU performs a PFN lookup to service a guest fault and the backing -guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be -consumed from guest_memfd, regardless of whether it is a shared or a private -fault. +When the KVM MMU performs a PFN lookup to service a guest fault, the fault will +always be consumed from guest_memfd, regardless of whether it is a shared or a +private fault (unless in-place conversion is disabled and the backing +guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP flag set). See KVM_SET_USER_MEMORY_REGION2 for additional details.virt/kvm/guest_memfd.c +0
-0virt/kvm/guest_memfd.c int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · line 750
Quoted 6 lines
virt/kvm/guest_memfd.c +1
-1@@ -746,7 +746,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,*/ WRITE_ONCE (slot->gmem.file, file); slot->gmem.pgoff = start; -if (kvm_gmem_supports_mmap (inode)) +if (gmem_in_place_conversion ||kvm_gmem_supports_mmap (inode)) slot->flags |=KVM_MEMSLOT_GMEM_ONLY ;I like this change, which actually enforces in-place conversion -- when
gmem_in_place_conversionis true, if userspace setsslot->userspace_addrto a different backend, the host and guest will no longer be able to access the same backend.
Quoted 6 lines
However, since
gmem_in_place_conversionis globally and statically specified, it essentially disables the coexistence of VMs using out-of-place conversions whengmem_in_place_conversionis true. Previously, VMs using out-of-place conversions could still boot successfully as long as they switched to using per-gmem memory attributes. Is this change intended?
Yes, this is intended. gmem_in_place_conversion is intended to be global for the entire host.
Quoted 5 lines
On a separate note, is
kvm_gmem_supports_mmap()still a necessary requirement for settingKVM_MEMSLOT_GMEM_ONLY? Whengmem_in_place_conversionis false, if userspace mmap()s the gmem but setsslot->userspace_addrto a different backend, should theKVM_MEMSLOT_GMEM_ONLYflag still be set?
Before this patch, requesting an MMAP-able gmem will set KVM_MEMSLOT_GMEM_ONLY, so if we change that behavior, we'd be breaking the existing contract.
If userspace requests an MMAP-able gmem and then sets up slot->userspace_addr to something else, I think that should be considered user error? MMAP-able => use gmem for all faults has been a thing since before this patch.
Just like how there's no good way to check and maintain the validity of the mapped memory behind slot->userspace_addr in relation to the provided guest_memfd, the mapping can't be used to determine whether to set KVM_MEMSLOT_GMEM_ONLY either.
Quoted 7 lines
Quoted 7 lines
virt/kvm/guest_memfd.c +0
-1xa_store_range (&f->bindings, start, end -1 , slot,GFP_KERNEL ); --2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 12/45] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
In reply to Ackerley TngQuoted 20 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
If a
guest_memfdmemslot is created but theguest_memfddoes not have theGUEST_MEMFD_FLAG_MMAPset, KVM still fulfils guest faults by looking up the memslot'suserspace_addr.Set
KVM_MEMSLOT_GMEM_ONLYif in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion,guest_memfdwill be the only backing memory for the memslot.No validation is performed to require
userspace_addrto be a mapping from the associatedguest_memfdbecause even after validation, userspace is free to remap something else at the provideduserspace_addr.
userspace_addrwill still be used by functions likekvm_read_guest(), and ifuserspace_addrdoes not match up with the corresponding memory in the memslot'sguest_memfd(whetheruserspace_addrpoints to the wrong offset or some non-guest_memfd memory, etc), that is a user error.
Quoted 7 lines
Requiring both shared and private memory to come from the only associated
guest_memfdsimplifies invalidation in stage 2 page tables. On aPUNCH_HOLEoperation on aguest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the givenguest_memfd.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Re: [PATCH v12 12/45] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled
In reply to Ackerley TngQuoted 20 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
If a
guest_memfdmemslot is created but theguest_memfddoes not have theGUEST_MEMFD_FLAG_MMAPset, KVM still fulfils guest faults by looking up the memslot'suserspace_addr.Set
KVM_MEMSLOT_GMEM_ONLYif in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion,guest_memfdwill be the only backing memory for the memslot.No validation is performed to require
userspace_addrto be a mapping from the associatedguest_memfdbecause even after validation, userspace is free to remap something else at the provideduserspace_addr.
userspace_addrwill still be used by functions likekvm_read_guest(), and ifuserspace_addrdoes not match up with the corresponding memory in the memslot'sguest_memfd(whetheruserspace_addrpoints to the wrong offset or some non-guest_memfd memory, etc), that is a user error.
Quoted 8 lines
Requiring both shared and private memory to come from the only associated
guest_memfdsimplifies invalidation in stage 2 page tables. On aPUNCH_HOLEoperation on aguest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the givenguest_memfd.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Yeah, that matches what we discussed in the call
Signature · 3 lines
Cheers, David
PATCH v12 13/45 KVM: guest_memfd: Pass mapping type filter to invalidation helper
In reply to Ackerley TngAccept the mapping type filter as a parameter in the invalidation start helper instead of querying it internally. This allows callers to specify which mappings (shared, private, or both) should be invalidated.
In the next patch, the conversion process will use this new parameter to invalidate mappings only when they're different from the target state of the conversion, i.e. invalidate only shared mappings on a shared to private conversion and not both.
No functional change intended.
--- virt/kvm/guest_memfd.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
virt/kvm/guest_memfd.c +9 -5
@@ -192,13 +192,13 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, }static void kvm_gmem_invalidate_start (struct inode *inode,pgoff_t start, -pgoff_t end) +pgoff_t end, +enum kvm_gfn_range_filter filter) { -enum kvm_gfn_range_filter attr_filter =kvm_gmem_get_all_gfns_filter (inode);struct gmem_file *f;kvm_gmem_for_each_file (f, inode) -__kvm_gmem_invalidate_start (f, start, end, attr_filter); +__kvm_gmem_invalidate_start (f, start, end, filter); } static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
@@ -224,6 +224,7 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,static long kvm_gmem_punch_hole (struct inode *inode,loff_t offset,loff_t len) { +enum kvm_gfn_range_filter filter =kvm_gmem_get_all_gfns_filter (inode);pgoff_t start = offset >>PAGE_SHIFT ;pgoff_t end = (offset + len) >>PAGE_SHIFT ;
@@ -233,7 +234,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)*/ filemap_invalidate_lock (inode->i_mapping); -kvm_gmem_invalidate_start (inode, start, end); +kvm_gmem_invalidate_start (inode, start, end, filter);truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 );
@@ -522,6 +523,8 @@ static int kvm_gmem_migrate_folio(struct address_space *mapping,static int kvm_gmem_error_folio (struct address_space *mapping,struct folio *folio) { +struct inode *inode = mapping->host; +enum kvm_gfn_range_filter filter;pgoff_t start, end;filemap_invalidate_lock_shared (mapping);
@@ -529,7 +532,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol start = folio->index; end = start +folio_nr_pages (folio); -kvm_gmem_invalidate_start (mapping->host, start, end); + filter =kvm_gmem_get_all_gfns_filter (inode); +kvm_gmem_invalidate_start (inode, start, end, filter);/* * Do not truncate the range, what action is taken in response to the
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 13/45] KVM: guest_memfd: Pass mapping type filter to invalidation helper
Quoted 7 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
Accept the mapping type filter as a parameter in the invalidation start helper instead of querying it internally. This allows callers to specify which mappings (shared, private, or both) should be invalidated.
Quoted 8 lines
In the next patch, the conversion process will use this new parameter to invalidate mappings only when they're different from the target state of the conversion, i.e. invalidate only shared mappings on a shared to private conversion and not both.
No functional change intended.
Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 66 lines
--- virt/kvm/guest_memfd.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)virt/kvm/guest_memfd.c +9
-6@@ -192,13 +192,13 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, }static void kvm_gmem_invalidate_start (struct inode *inode,pgoff_t start, -pgoff_t end) +pgoff_t end, +enum kvm_gfn_range_filter filter) { -enum kvm_gfn_range_filter attr_filter =kvm_gmem_get_all_gfns_filter (inode);struct gmem_file *f;kvm_gmem_for_each_file (f, inode) -__kvm_gmem_invalidate_start (f, start, end, attr_filter); +__kvm_gmem_invalidate_start (f, start, end, filter); } static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,@@ -224,6 +224,7 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,static long kvm_gmem_punch_hole (struct inode *inode,loff_t offset,loff_t len) { +enum kvm_gfn_range_filter filter =kvm_gmem_get_all_gfns_filter (inode);pgoff_t start = offset >>PAGE_SHIFT ;pgoff_t end = (offset + len) >>PAGE_SHIFT ;@@ -233,7 +234,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)*/ filemap_invalidate_lock (inode->i_mapping); -kvm_gmem_invalidate_start (inode, start, end); +kvm_gmem_invalidate_start (inode, start, end, filter);truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 );@@ -522,6 +523,8 @@ static int kvm_gmem_migrate_folio(struct address_space *mapping,static int kvm_gmem_error_folio (struct address_space *mapping,struct folio *folio) { +struct inode *inode = mapping->host; +enum kvm_gfn_range_filter filter;pgoff_t start, end;filemap_invalidate_lock_shared (mapping);@@ -529,7 +532,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol start = folio->index; end = start +folio_nr_pages (folio); -kvm_gmem_invalidate_start (mapping->host, start, end); + filter =kvm_gmem_get_all_gfns_filter (inode); +kvm_gmem_invalidate_start (inode, start, end, filter);/* * Do not truncate the range, what action is taken in response to the -- 2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 13/45] KVM: guest_memfd: Pass mapping type filter to invalidation helper
In reply to Ackerley TngQuoted 6 lines · On 8/31/2026 8:25 AM, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
Accept the mapping type filter as a parameter in the invalidation start helper instead of querying it internally. This allows callers to specify which mappings (shared, private, or both) should be invalidated.
Quoted 8 lines
In the next patch, the conversion process will use this new parameter to invalidate mappings only when they're different from the target state of the conversion, i.e. invalidate only shared mappings on a shared to private conversion and not both.
No functional change intended.
Signed-off-by Ackerley Tng <[email protected]>
Re: [PATCH v12 13/45] KVM: guest_memfd: Pass mapping type filter to invalidation helper
In reply to Ackerley TngQuoted 6 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
Accept the mapping type filter as a parameter in the invalidation start helper instead of querying it internally. This allows callers to specify which mappings (shared, private, or both) should be invalidated.
Quoted 8 lines
In the next patch, the conversion process will use this new parameter to invalidate mappings only when they're different from the target state of the conversion, i.e. invalidate only shared mappings on a shared to private conversion and not both.
No functional change intended.
Signed-off-by Ackerley Tng <[email protected]>
Signature · 3 lines
Cheers, David
PATCH v12 14/45 KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
In reply to Ackerley TngAdd a new ioctl (and matching struct), KVM_SET_MEMORY_ATTRIBUTES2, using the same base ioctl number (0xd2), but with R/W semantics for the kernel instead of just read semantics. "Officially" documenting that KVM writes to the payload will allow KVM to support partial/incremental conversions, instead of all-or-nothing updates (which requires complex unwinding), by recording the failing offset if an error occurs.
Opportunistically add a new struct as well, even though KVM could squeeze the error offset into "struct kvm_memory_attributes", as there's no cost to doing so in practice. Pad the struct with a pile of extra space to try and avoid ending up with "struct kvm_memory_attributes3" in the future. Use the same layout for the fields common to version 1 of the struct, e.g. to ease upgrading userspace, and to provide flexibility if KVM ever adds support for KVM_SET_MEMORY_ATTRIBUTES2 at VM scope.
Introduce KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES to advertise the availability of the KVM_SET_MEMORY_ATTRIBUTES2 ioctl.
Update the KVM API documentation to define the new ioctl and its behavior, and add the necessary UAPI definitions and capability checks.
The process of setting memory attributes has a clear point of no return because, for CoCo VMs, zapping stage 2 page tables is a destructive operation. Unlike regular VMs, where re-faulting pages into the stage 2 page tables merely incurs a performance penalty, CoCo guests must (re-):accept pages after every fault. To preserve CoCo security guarantees, guests will not accept pages they did not explicitly request faults for. Consequently, during memory conversions, any operation that could cause the process to abort must be completed before the stage 2 page tables are zapped.
Zap only the ranges that are not already in the requested state to avoid inadvertently destroying (CoCo) data. ARM CCA guests will try to mark the entire DRAM as private at boot. If there are no shared pages at all, the to-private conversion can be skipped, but the existence of a single shared page would require the conversion process to proceed, and if it proceeds, zapping both shared and private pages would destroy data and break the guest.
--- Documentation/virt/kvm/api.rst | 71 +++++++++++++++++++++++- include/uapi/linux/kvm.h | 15 ++++++ virt/kvm/guest_memfd.c | 119 +++++++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 23 +++++--- 4 files changed, 219 insertions(+), 9 deletions(-)
Documentation/virt/kvm/api.rst +69 -2
@@ -117,7 +117,7 @@ description: x86 includes both i386 and x86_64. Type: - system, vm, or vcpu. + system, vm, vcpu or guest_memfd. Parameters: what parameters are accepted by the ioctl.
@@ -6385,7 +6385,9 @@ When mapping a gfn into the guest, guest faults are always serviced from guest_memfd regardless of whether memory is shared or private. KVM determines shared vs. private based on the state in guest_memfd, which is the sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to -find out more about the creation-time shared/private status. +find out more about the creation-time shared/private status. Userspace can +control whether memory is shared/private by toggling +KVM_MEMORY_ATTRIBUTE_PRIVATE via :ref:`KVM_SET_MEMORY_ATTRIBUTES2` as needed. userspace_addr is expected to be the mmap()-ed address corresponding to the right offset within the guest_memfd. Any mismatch between userspace_addr and
@@ -6404,6 +6406,8 @@ S390: Returns -EINVAL if the VM has the KVM_VM_S390_UCONTROL flag set. Returns -EINVAL if called on a protected VM. +.. _KVM_SET_MEMORY_ATTRIBUTES: + 4.141 KVM_SET_MEMORY_ATTRIBUTES -------------------------------
@@ -6440,6 +6444,8 @@ the state of a gfn/page as needed. The "flags" field is reserved for future extensions and must be '0'. +See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES2`. + .. _KVM_CREATE_GUEST_MEMFD: 4.142 KVM_CREATE_GUEST_MEMFD
@@ -6599,6 +6605,67 @@ KVM_S390_KEYOP_SSKE Sets the storage key for the guest address ``guest_addr`` to the key specified in ``key``, returning the previous value in ``key``. +.. _KVM_SET_MEMORY_ATTRIBUTES2: + +4.145 KVM_SET_MEMORY_ATTRIBUTES2 +--------------------------------- + +:Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES +:Architectures: all +:Type: guest_memfd ioctl +:Parameters: struct kvm_memory_attributes2 (in) +:Returns: 0 on success, <0 on error + +Errors: + + ========== =============================================================== + EINVAL The specified `offset` or `size` was invalid (e.g. not + page aligned, causes an overflow, or size is zero). + EFAULT The parameter address was invalid. + ENOMEM Ran out of memory trying to track private/shared state + ========== =============================================================== + +KVM_SET_MEMORY_ATTRIBUTES2 is an extension to +KVM_SET_MEMORY_ATTRIBUTES that supports returning (writing) values to +userspace. The original (pre-extension) fields are shared with +KVM_SET_MEMORY_ATTRIBUTES identically. + +Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. + +:: + + struct kvm_memory_attributes2 { + union { + __u64 address; + __u64 offset; + }; + __u64 size; + __u64 attributes; + __u64 flags; + __u64 reserved[12]; + }; + + #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) + +Set attributes for a range of offsets within a guest_memfd to +KVM_MEMORY_ATTRIBUTE_PRIVATE to limit the specified guest_memfd backed +memory range for guest use. Even if KVM_CAP_GUEST_MEMFD_MMAP is +supported, after a successful call to set +KVM_MEMORY_ATTRIBUTE_PRIVATE, the requested range will not be mappable +into host userspace and will only be mappable by the guest. + +To allow the range to be mappable into host userspace again, call +KVM_SET_MEMORY_ATTRIBUTES2 on the guest_memfd again with +KVM_MEMORY_ATTRIBUTE_PRIVATE unset. + +KVM does not directly manipulate the memory contents of pages during +attribute updates. However, the process of setting these attributes, +which includes operations such as unmapping pages from the host or +stage-2 page tables, may result in side effects on memory contents +that vary across different trusted firmware implementations. + +See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`. + 4.145 KVM_PPC_GET_COMPAT_CAPS ----------------------------- :Capability: KVM_CAP_PPC_COMPAT_CAPS
include/uapi/linux/kvm.h +15 -0
@@ -999,6 +999,7 @@ struct kvm_enable_cap {#define KVM_CAP_S390_HPAGE_2G 249#define KVM_CAP_PPC_COMPAT_CAPS 250#define KVM_CAP_ARM_PMU_V3_STRICT 251 +#define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 252struct kvm_irq_routing_irqchip {__u32 irqchip;
@@ -1653,6 +1654,20 @@ struct kvm_memory_attributes {__u64 flags; }; +/* Available with KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES */ +#define KVM_SET_MEMORY_ATTRIBUTES2 _IOWR(KVMIO, 0xd2, struct kvm_memory_attributes2) + +struct kvm_memory_attributes2 { +union { +__u64 address; +__u64 offset; + }; +__u64 size; +__u64 attributes; +__u64 flags; +__u64 reserved[12 ]; +}; +#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)#define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd)
virt/kvm/guest_memfd.c +119 -0
130 lines not on this page · raw message
virt/kvm/kvm_main.c +16 -7
@@ -2423,18 +2423,22 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, }#endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ +#ifdef kvm_arch_has_private_mem +static u64 kvm_supports_private_mem (struct kvm *kvm) +{ +return !kvm ||kvm_arch_has_private_mem (kvm); +} +#else +#define kvm_supports_private_mem (kvm) false +#endif +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_vm_mem_attributes (struct kvm *kvm) { -#ifdef kvm_arch_has_private_mem -if (gmem_in_place_conversion) +if (gmem_in_place_conversion || !kvm_supports_private_mem (kvm))return 0 ; -if (!kvm ||kvm_arch_has_private_mem (kvm)) -return KVM_MEMORY_ATTRIBUTE_PRIVATE ; -#endif - -return 0 ; +return KVM_MEMORY_ATTRIBUTE_PRIVATE ; }/*
@@ -4976,6 +4980,11 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)return 1 ;case KVM_CAP_GUEST_MEMFD_FLAGS :return kvm_gmem_get_supported_flags (kvm); +case KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES : +if (!gmem_in_place_conversion || !kvm_supports_private_mem (kvm)) +return 0 ; + +return KVM_MEMORY_ATTRIBUTE_PRIVATE ;#endif default :break ;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 14/45] KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
Hi Ackerley,
Quoted 2 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
...
Quoted 3 lines
+..
_KVM_SET_MEMORY_ATTRIBUTES2:
+
+4.145KVM_SET_MEMORY_ATTRIBUTES2
This section number collides with the existing 4.145 (KVM_PPC_GET_COMPAT_CAPS) below, so api.rst ends up with two 4.145 sections. One of them needs renumbering.
...
Quoted 1 line
4.145
KVM_PPC_GET_COMPAT_CAPS
Cheers,
/fuad
Re: [PATCH v12 14/45] KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
Quoted 11 lines · Fuad Tabba
Hi Ackerley,
Quoted 2 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
...
Quoted 3 lines
+..
_KVM_SET_MEMORY_ATTRIBUTES2:
+
+4.145KVM_SET_MEMORY_ATTRIBUTES2
Quoted 5 lines
This section number collides with the existing 4.145 (
KVM_PPC_GET_COMPAT_CAPS) below, soapi.rstends up with two 4.145 sections. One of them needs renumbering....
I'll add KVM_SET_MEMORY_ATTRIBUTES2 as 4.146 below KVM_PPC_GET_COMPAT_CAPS. Thanks, good catch!
Quoted 4 lines
Quoted 1 line
4.145
KVM_PPC_GET_COMPAT_CAPSCheers,
/fuad
PATCH v12 15/45 KVM: guest_memfd: Ensure pages are not in use before conversion
In reply to Ackerley TngWhen converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.
guest_memfd checks for any outstanding references to determine whether a page is still in use. The only expected references after unmapping the range requested for conversion are those that are held by guest_memfd itself.
Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.
Update documentation to document the error_offset field and the possible -EAGAIN error.
--- Documentation/virt/kvm/api.rst | 19 +++++++++-- include/uapi/linux/kvm.h | 3 +- virt/kvm/guest_memfd.c | 77 +++++++++++++++++++++++++++++++++++++++--- 3 files changed, 91 insertions(+), 8 deletions(-)
Documentation/virt/kvm/api.rst +17 -2
@@ -6613,7 +6613,7 @@ KVM_S390_KEYOP_SSKE :Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES :Architectures: all :Type: guest_memfd ioctl -:Parameters: struct kvm_memory_attributes2 (in) +:Parameters: struct kvm_memory_attributes2 (in/out) :Returns: 0 on success, <0 on error Errors:
@@ -6622,6 +6622,8 @@ Errors: EINVAL The specified `offset` or `size` was invalid (e.g. not page aligned, causes an overflow, or size is zero). EFAULT The parameter address was invalid. + EAGAIN Some page within requested range had unexpected refcounts. The + offset of the page will be returned in `error_offset`. ENOMEM Ran out of memory trying to track private/shared state ========== ===============================================================
@@ -6635,6 +6637,7 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. :: struct kvm_memory_attributes2 { + /* in */ union { __u64 address; __u64 offset;
@@ -6642,7 +6645,9 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. __u64 size; __u64 attributes; __u64 flags; - __u64 reserved[12]; + /* out */ + __u64 error_offset; + __u64 reserved[11]; }; #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)
@@ -6664,6 +6669,16 @@ which includes operations such as unmapping pages from the host or stage-2 page tables, may result in side effects on memory contents that vary across different trusted firmware implementations. +If this ioctl returns -EAGAIN, the offset of the page with unexpected +refcounts will be returned in ``error_offset``. This can occur if +there are transient refcounts on the pages, taken by other parts of +the kernel. + +Userspace is expected to figure out how to remove all known refcounts +on the shared pages, such as refcounts taken by get_user_pages(), and +try the ioctl again. A possible source of these long term refcounts is +if the guest_memfd memory was pinned in IOMMU page tables. + See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`. 4.145 KVM_PPC_GET_COMPAT_CAPS
include/uapi/linux/kvm.h +2 -1
@@ -1665,7 +1665,8 @@ struct kvm_memory_attributes2 {__u64 size;__u64 attributes;__u64 flags; -__u64 reserved[12 ]; +__u64 error_offset; +__u64 reserved[11 ]; };#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)
virt/kvm/guest_memfd.c +72 -5
105 lines not on this page · raw message
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 15/45] KVM: guest_memfd: Ensure pages are not in use before conversion
Hi Ackerley,
Quoted 119 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
When converting memory to private in
guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.
guest_memfdchecks for any outstanding references to determine whether a page is still in use. The only expected references after unmapping the range requested for conversion are those that are held byguest_memfditself.Update the
kvm_memory_attributes2structure to include anerror_offsetfield. This allows KVM to report the exact offset where a conversion failed. If the safety check fails, return-EAGAINand copy theerror_offsetback to userspace so that it can potentially retry the operation or handle the failure gracefully.Update documentation to document the
error_offsetfield and the possible-EAGAINerror.Suggested-by David Hildenbrand <[email protected]>Co-developed-by Vishal Annapurve <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>Reviewed-by Fuad Tabba <[email protected]>Tested-by Shivank Garg <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- Documentation/virt/kvm/api.rst | 19 +++++++++-- include/uapi/linux/kvm.h | 3 +- virt/kvm/guest_memfd.c | 77 +++++++++++++++++++++++++++++++++++++++--- 3 files changed, 91 insertions(+), 8 deletions(-)Documentation/virt/kvm/api.rst +17
-2@@ -6613,7 +6613,7 @@ KVM_S390_KEYOP_SSKE :Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES :Architectures: all :Type: guest_memfd ioctl -:Parameters: struct kvm_memory_attributes2 (in) +:Parameters: struct kvm_memory_attributes2 (in/out) :Returns: 0 on success, <0 on error Errors:@@ -6622,6 +6622,8 @@ Errors: EINVAL The specified `offset` or `size` was invalid (e.g. not page aligned, causes an overflow, or size is zero). EFAULT The parameter address was invalid. + EAGAIN Some page within requested range had unexpected refcounts. The + offset of the page will be returned in `error_offset`. ENOMEM Ran out of memory trying to track private/shared state ========== ===============================================================@@ -6635,6 +6637,7 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. :: struct kvm_memory_attributes2 { + /* in */ union { __u64 address; __u64 offset;@@ -6642,7 +6645,9 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. __u64 size; __u64 attributes; __u64 flags; - __u64 reserved[12]; + /* out */ + __u64 error_offset; + __u64 reserved[11]; }; #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)@@ -6664,6 +6669,16 @@ which includes operations such as unmapping pages from the host or stage-2 page tables, may result in side effects on memory contents that vary across different trusted firmware implementations. +If this ioctl returns -EAGAIN, the offset of the page with unexpected +refcounts will be returned in ``error_offset``. This can occur if +there are transient refcounts on the pages, taken by other parts of +the kernel. + +Userspace is expected to figure out how to remove all known refcounts +on the shared pages, such as refcounts taken by get_user_pages(), and +try the ioctl again. A possible source of these long term refcounts is +if the guest_memfd memory was pinned in IOMMU page tables. + See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`. 4.145 KVM_PPC_GET_COMPAT_CAPSinclude/uapi/linux/kvm.h +2
-1@@ -1665,7 +1665,8 @@ struct kvm_memory_attributes2 {__u64 size;__u64 attributes;__u64 flags; -__u64 reserved[12 ]; +__u64 error_offset; +__u64 reserved[11 ]; };#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)virt/kvm/guest_memfd.c +5
-0@@ -538,8 +538,46 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,return mas_preallocate (mas,xa_mk_value (attributes),GFP_KERNEL ); } +static bool kvm_gmem_has_outstanding_references (struct inode *inode, +pgoff_t start,size_t nr_pages, +pgoff_t *err_index) +{ +struct address_space *mapping = inode->i_mapping;
virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 555
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+pgoff_t last = start + nr_pages -1 ; +bool has_outstanding = false; +struct folio_batch fbatch; +pgoff_t next; +int i; + +folio_batch_init (&fbatch); + + next = start; +while (has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) {
has_outstanding starts as false, so the loop never runs and the function always returns false. The outstanding-reference check is dead at this patch, so a to-private conversion would not be rejected even when a page still has an outstanding reference.
It's fixed later in "KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check", which changes the condition to !has_outstanding. I think that fix belongs in this patch, so the check works when it is introduced and the series bisects cleanly.
Missed it on the previous review, but with this fixed it still holds.
Cheers,
/fuad
Quoted 95 lines
virt/kvm/guest_memfd.c +57
-6+for (i =0 ; i <folio_batch_count (&fbatch); ++i) { +struct folio *folio = fbatch.folios[i]; + +/* +* Outstanding references are anything other than those +* from the page cache, plus 1 temporary reference held +* by filemap_get_folios() in the folio batch. +*/ +if (folio_ref_count (folio) !=folio_nr_pages (folio) +1 ) { + has_outstanding = true; + *err_index =max (start, folio->index); +break ; + } + } + +folio_batch_release (&fbatch); +cond_resched (); + } + +return has_outstanding; +} +static int __kvm_gmem_set_attributes (struct inode *inode,pgoff_t start, -size_t nr_pages,uint64_t attrs) +size_t nr_pages,uint64_t attrs, +pgoff_t *err_index) {bool to_private = attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ;struct address_space *mapping = inode->i_mapping;@@ -556,8 +594,28 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,mas_init (&mas, mt, start); r =kvm_gmem_mas_preallocate (&mas, attrs, start, nr_pages); -if (r) +if (r) { + *err_index = start; goto out; + } + +if (to_private) { +/* +* Forcefully unmap the pages from all userspace page tables, +* and then verify there are no outstanding references, e.g. +* acquired via GUP or similar. Tell userspace to try again if +* there are outstanding references and hope that whatever has +* pinned the page will put its reference "soon". +*/ +unmap_mapping_pages (mapping, start, nr_pages, false); + +if (kvm_gmem_has_outstanding_references (inode, start, nr_pages, + err_index)) { +mas_destroy (&mas); + r = -EAGAIN ; + goto out; + } + }/* * From this point on guest_memfd has performed necessary @@ -578,9 +636,10 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp)struct gmem_file *f = file->private_data;struct inode *inode =file_inode (file);struct kvm_memory_attributes2 attrs; +pgoff_t err_index;size_t nr_pages;pgoff_t index; -int i; +int i, r;if (copy_from_user (&attrs, argp,sizeof (attrs)))return -EFAULT ;@@ -606,8 +665,16 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp) nr_pages = attrs.size >>PAGE_SHIFT ; index = attrs.offset >>PAGE_SHIFT ; -return __kvm_gmem_set_attributes (inode, index, nr_pages, - attrs.attributes); + r =__kvm_gmem_set_attributes (inode, index, nr_pages, attrs.attributes, + &err_index); +if (r) { + attrs.error_offset = ((uint64_t )err_index) <<PAGE_SHIFT ; + +if (copy_to_user (argp, &attrs,sizeof (attrs))) +return -EFAULT ; + } + +return r; } static long kvm_gmem_ioctl(struct file *file, unsigned int ioctl, --2.55.0.897.gb25b4bd76c-goog
Sean Christopherson
maintainerRe: [PATCH v12 15/45] KVM: guest_memfd: Ensure pages are not in use before conversion
Quoted 29 lines · Fuad Tabba
Quoted 13 lines
virt/kvm/guest_memfd.c +5
-0@@ -538,8 +538,46 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,return mas_preallocate (mas,xa_mk_value (attributes),GFP_KERNEL ); } +static bool kvm_gmem_has_outstanding_references (struct inode *inode, +pgoff_t start,size_t nr_pages, +pgoff_t *err_index) +{ +struct address_space *mapping = inode->i_mapping;virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 555
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+pgoff_t last = start + nr_pages -1 ; +bool has_outstanding = false; +struct folio_batch fbatch; +pgoff_t next; +int i; + +folio_batch_init (&fbatch); + + next = start; +while (has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) {
has_outstandingstarts as false, so the loop never runs and the function always returns false. The outstanding-reference check is dead at this patch, so a to-private conversion would not be rejected even when a page still has an outstanding reference.
Quoted 4 lines
It's fixed later in "KVM:
guest_memfd: Handlelru_addfbatch refcounts during conversion safety check", which changes the condition to !has_outstanding. I think that fix belongs in this patch, so the check works when it is introduced and the series bisects cleanly.
Why even bother with has_outstanding? Avoiding it requires copy+pasting folio_batch_release(), but it's less code and IMO the end result is a lot easier to follow:
struct address_space *mapping = inode->i_mapping;
pgoff_t last = start + nr_pages - 1;
struct folio_batch fbatch;
pgoff_t next;
int i;
folio_batch_init(&fbatch);
next = start;
while (filemap_get_folios(mapping, &next, last, &fbatch)) {
for (i = 0; i < folio_batch_count(&fbatch); ++i) {
struct folio *folio = fbatch.folios[i];
/*
* Outstanding references are anything other than those
* from the page cache, plus 1 temporary reference held
* by filemap_get_folios() in the folio batch.
*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1) {
*err_index = max(start, folio->index);
folio_batch_release(&fbatch);
return true;
}
}
folio_batch_release(&fbatch);
cond_resched();
}
return false;and then we end up with:
enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
struct address_space *mapping = inode->i_mapping;
pgoff_t last = start + nr_pages - 1;
struct folio_batch fbatch;
pgoff_t next;
int i;
folio_batch_init(&fbatch);
next = start;
while (filemap_get_folios(mapping, &next, last, &fbatch)) {
for (i = 0; i < folio_batch_count(&fbatch); ++i) {
struct folio *folio = fbatch.folios[i];
if (__folio_has_outstanding_references(folio, &drained)) {
*err_index = max(start, folio->index);
folio_batch_release(&fbatch);
return true;
}
}
folio_batch_release(&fbatch);
cond_resched();
}
return false;Sean Christopherson
maintainerRe: [PATCH v12 15/45] KVM: guest_memfd: Ensure pages are not in use before conversion
Quoted 30 lines · Sean Christopherson
Quoted 29 lines · Fuad Tabba
Quoted 13 lines
virt/kvm/guest_memfd.c +5
-0@@ -538,8 +538,46 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,return mas_preallocate (mas,xa_mk_value (attributes),GFP_KERNEL ); } +static bool kvm_gmem_has_outstanding_references (struct inode *inode, +pgoff_t start,size_t nr_pages, +pgoff_t *err_index) +{ +struct address_space *mapping = inode->i_mapping;virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 555
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+pgoff_t last = start + nr_pages -1 ; +bool has_outstanding = false; +struct folio_batch fbatch; +pgoff_t next; +int i; + +folio_batch_init (&fbatch); + + next = start; +while (has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) {
has_outstandingstarts as false, so the loop never runs and the function always returns false. The outstanding-reference check is dead at this patch, so a to-private conversion would not be rejected even when a page still has an outstanding reference.
Quoted 4 lines
Quoted 4 lines
It's fixed later in "KVM:
guest_memfd: Handlelru_addfbatch refcounts during conversion safety check", which changes the condition to !has_outstanding. I think that fix belongs in this patch, so the check works when it is introduced and the series bisects cleanly.
Wait, why are there even separate patches for this? For all intents and purposes, "Ensure pages are not in use before conversion" introduces a bug and then the bug is fixed by "Handle lru_add fbatch refcounts during conversion safety check". Just don't introduce the bug.
I also recommend splitting the export of lru_cache_drain_for_folio() to its own patch so that it can be more easily Acked by mm/ folks. If we want to squash it with the KVM change, then that's trivial to do when applying.
Quoted 63 lines
Why even bother with
has_outstanding? Avoiding it requires copy+pastingfolio_batch_release(), but it's less code and IMO the end result is a lot easier to follow:struct address_space *mapping = inode->i_mapping; pgoff_t last = start + nr_pages - 1; struct folio_batch fbatch; pgoff_t next; int i; folio_batch_init(&fbatch); next = start; while (filemap_get_folios(mapping, &next, last, &fbatch)) { for (i = 0; i < folio_batch_count(&fbatch); ++i) { struct folio *folio = fbatch.folios[i]; /* * Outstanding references are anything other than those * from the page cache, plus 1 temporary reference held * by filemap_get_folios() in the folio batch. */ if (folio_ref_count(folio) != folio_nr_pages(folio) + 1) { *err_index = max(start, folio->index); folio_batch_release(&fbatch); return true; } } folio_batch_release(&fbatch); cond_resched(); } return false;and then we end up with:
enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED; struct address_space *mapping = inode->i_mapping; pgoff_t last = start + nr_pages - 1; struct folio_batch fbatch; pgoff_t next; int i; folio_batch_init(&fbatch); next = start; while (filemap_get_folios(mapping, &next, last, &fbatch)) { for (i = 0; i < folio_batch_count(&fbatch); ++i) { struct folio *folio = fbatch.folios[i]; if (__folio_has_outstanding_references(folio, &drained)) { *err_index = max(start, folio->index); folio_batch_release(&fbatch); return true; } } folio_batch_release(&fbatch); cond_resched(); } return false;
PATCH v12 16/45 KVM: guest_memfd: Call arch make_shared callback for to-shared conversion
In reply to Ackerley TngWhen doing in-place conversion from PRIVATE to SHARED, immediately inform arch code of the conversion for all allocated pages/folios, e.g. so that arch code can put hardware metadata tables in the correct state. Eagerly updating the table for to SHARED conversions avoids having to implement on-demand updates, e.g. when faulting in host userspace mappings. Skip the entire flow if the arch doesn't implement conversion callbacks, as getting folios from the filemap is noticeably expensive, especially when converting large chunks of memory.
Deliberately don't eagerly update the metadata table on conversions from SHARED to PRIVATE, because assigning a page to a VM (versus "returning" it to the host) requires the exact GFN associated with the page, i.e would require walking the memslot bindings. And because KVM *must* do on-demand metadata updates when getting a PFN for KVM-internal usage, as that's the only time a relevant memslot binding is guaranteed to exist.
Note! Inform arch code of the conversion within the protection of the invalidation sequence, to ensure that any existing mappings are dropped before hardware is updated, and to ensure that new mappings can't be established until after the conversion is complete.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/x86.c | 5 +++++ include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 42 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 50 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -150,7 +150,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT KVM_X86_OP_OPTIONAL_RET0(gmem_make_private)#endif -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM ) KVM_X86_OP_OPTIONAL(gmem_make_shared)#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1732,7 +1732,7 @@ struct kvm_x86_ops {int (*gmem_make_private)(struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,kvm_pfn_t nr_pages);#endif -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM )void (*gmem_make_shared)(kvm_pfn_t pfn,kvm_pfn_t nr_pages);#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/x86.c +5 -0
@@ -10653,6 +10653,11 @@ int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, {return kvm_x86_call (gmem_make_private)(kvm, gfn, pfn, nr_pages); } + +void kvm_arch_gmem_make_shared (kvm_pfn_t pfn,kvm_pfn_t nr_pages) +{ +kvm_x86_call (gmem_make_shared)(pfn, nr_pages); +}#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
include/linux/kvm_host.h +1 -0
@@ -2610,6 +2610,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,int kvm_arch_gmem_make_private (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,kvm_pfn_t nr_pages); +void kvm_arch_gmem_make_shared (kvm_pfn_t pfn,kvm_pfn_t nr_pages);#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT #define kvm_arch_has_gmem_convert () false#endif
virt/kvm/guest_memfd.c +42 -0
55 lines not on this page · raw message
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 17/45 KVM: guest_memfd: Return early if range already has requested attributes
In reply to Ackerley TngProvide a function to check that a range has given attributes.
Optimize setting memory attributes by returning early if all pages in the requested range already have the requested attributes.
--- virt/kvm/guest_memfd.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
virt/kvm/guest_memfd.c +22 -1
@@ -98,6 +98,24 @@ static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)return !kvm_gmem_is_private_mem (inode, index); } +static bool kvm_gmem_range_has_attributes (struct inode *inode, +pgoff_t index,size_t nr_pages, +u64 attributes) +{ +struct maple_tree *mt = &GMEM_I (inode)->attributes; +pgoff_t end = index + nr_pages -1 ; +void *entry; + +lockdep_assert (mt_lock_is_held (mt)); + +mt_for_each (mt, entry, index, end) { +if (kvm_gmem_get_attributes (inode, entry) != attributes) +return false; + } + +return true; +} +/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest.
@@ -623,12 +641,15 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,pgoff_t end = start + nr_pages;struct maple_tree *mt;struct ma_state mas; -int r; +int r =0 ; mt = &gi->attributes;filemap_invalidate_lock (mapping); +if (kvm_gmem_range_has_attributes (inode, start, nr_pages, attrs)) + goto out; +mas_init (&mas, mt, start); r =kvm_gmem_mas_preallocate (&mas, attrs, start, nr_pages);if (r) {
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 17/45] KVM: guest_memfd: Return early if range already has requested attributes
Quoted 21 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
Provide a function to check that a range has given attributes.
Optimize setting memory attributes by returning early if all pages in the requested range already have the requested attributes.
Reviewed-by Fuad Tabba <[email protected]>Tested-by Shivank Garg <[email protected]>Reviewed-by Binbin Wu <[email protected]>Reviewed-by Xiaoyao Li <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- virt/kvm/guest_memfd.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)virt/kvm/guest_memfd.c +0
-0
virt/kvm/guest_memfd.c static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index) · line 103
Quoted 6 lines
virt/kvm/guest_memfd.c +3
-0@@ -98,6 +98,24 @@ static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)return !kvm_gmem_is_private_mem (inode, index); } +static bool kvm_gmem_range_has_attributes (struct inode *inode, +pgoff_t index,size_t nr_pages, +u64 attributes)
"has_attributes" could be confused with "has any attributes", not "has exactly these matching attributes and no others".
Anyhow, I'm out of puff for naming suggestions today and this gets the hob done ;) (kvm_gmem_range_has_matching_attributes is a bit mouthful)
Signature · 3 lines
Cheers, David
PATCH v12 18/45 KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
In reply to Ackerley TngA guest_memfd folio has no outstanding references if guest_memfd holds the only references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.
A folio will have outstanding references if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.
A folio may also have extra refcounts if it is on the mlock fbatch.
These two known "usages" of the folio are handled by calling lru_cache_drain_for_folio, which drains both the lru_add and mlock fbatches. After draining, if the refcount is still elevated, then there are truly outstanding references.
If the page may be dma pinned, DMA is using it and hence there are outstanding references. folio_maybe_dma_pinned() can have false positives, but that's only with a significant number of refcounts, at which point draining LRU is not going to move the needle - it can still be concluded that the folio has outstanding references.
If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it also has outstanding references.
Return true and exit early to avoid unnecessary draining in these 2 cases.
Provide a drain status to only drain once ever while processing a batch of folios.
--- mm/folio.c | 2 ++ virt/kvm/guest_memfd.c | 30 ++++++++++++++++++++++-------- 2 files changed, 24 insertions(+), 8 deletions(-)
mm/folio.c +2 -0
@@ -33,6 +33,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_types.h> #include "internal.h" #include "page_alloc.h"
@@ -926,6 +927,7 @@ void lru_cache_drain_for_folio(const struct folio *folio, *drained =LRU_CACHE_DRAINED_ALL ; } } +EXPORT_SYMBOL_FOR_KVM (lru_cache_drain_for_folio);atomic_t lru_disable_count =ATOMIC_INIT (0 );
virt/kvm/guest_memfd.c +22 -8
@@ -8,6 +8,7 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/swap.h> #include "kvm_mm.h" #include "guest_memfd.h"
@@ -556,10 +557,28 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,return mas_preallocate (mas,xa_mk_value (attributes),GFP_KERNEL ); } +static bool __folio_has_outstanding_references (struct folio *folio, +enum lru_cache_drained *drained) +{ +if (folio_maybe_dma_pinned (folio) ||folio_mapped (folio)) +return true; + +/* 1 reference held by filemap_get_folios() in the folio batch. */ +lru_cache_drain_for_folio (folio,1 , drained); + +/* +* Outstanding references are anything other than those from the page +* cache, plus 1 temporary reference held by filemap_get_folios() in the +* folio batch. +*/ +return folio_ref_count (folio) !=folio_nr_pages (folio) +1 ; +} +static bool kvm_gmem_has_outstanding_references (struct inode *inode,pgoff_t start,size_t nr_pages,pgoff_t *err_index) { +enum lru_cache_drained drained =LRU_CACHE_NOT_DRAINED ;struct address_space *mapping = inode->i_mapping;pgoff_t last = start + nr_pages -1 ;bool has_outstanding = false;
@@ -570,17 +589,12 @@ static bool kvm_gmem_has_outstanding_references(struct inode *inode,folio_batch_init (&fbatch); next = start; -while (has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) { +while (!has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) {for (i =0 ; i <folio_batch_count (&fbatch); ++i) {struct folio *folio = fbatch.folios[i]; -/* -* Outstanding references are anything other than those -* from the page cache, plus 1 temporary reference held -* by filemap_get_folios() in the folio batch. -*/ -if (folio_ref_count (folio) !=folio_nr_pages (folio) +1 ) { - has_outstanding = true; + has_outstanding =__folio_has_outstanding_references (folio, &drained); +if (has_outstanding) { *err_index =max (start, folio->index);break ; }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 18/45] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
Quoted 56 lines · On Sun, 30 Aug 2026, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
A
guest_memfdfolio has no outstanding references ifguest_memfdholds the only references on it. Any other references on the folio may indicate another user, andguest_memfdcannot convert it to private if there may be an existing host user.A folio will have outstanding references if it is present in a per-CPU
lru_addfbatch.guest_memfddoes not actually participate in LRU, but freshly-allocated folios are still added to thelru_addfbatch for batch LRU statistics processing.A folio may also have extra refcounts if it is on the mlock fbatch.
These two known "usages" of the folio are handled by calling
lru_cache_drain_for_folio, which drains both thelru_addand mlock fbatches. After draining, if the refcount is still elevated, then there are truly outstanding references.If the page may be dma pinned, DMA is using it and hence there are outstanding references.
folio_maybe_dma_pinned()can have false positives, but that's only with a significant number of refcounts, at which point draining LRU is not going to move the needle - it can still be concluded that the folio has outstanding references.If the page is still mapped after
guest_memfdtried to unmap it earlier in the conversion process, it also has outstanding references.Return true and exit early to avoid unnecessary draining in these 2 cases.
Provide a drain status to only drain once ever while processing a batch of folios.
Suggested-by David Hildenbrand <[email protected]>Reviewed-by Fuad Tabba <[email protected]>Reviewed-by Binbin Wu <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- mm/folio.c | 2 ++ virt/kvm/guest_memfd.c | 30 ++++++++++++++++++++++-------- 2 files changed, 24 insertions(+), 8 deletions(-)mm/folio.c +1
-0@@ -33,6 +33,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_types.h> #include "internal.h" #include "page_alloc.h"
mm/folio.c void lru_cache_drain_for_folio(const struct folio *folio, · line 933
Quoted 7 lines
mm/folio.c +1
-0@@ -926,6 +927,7 @@ void lru_cache_drain_for_folio(const struct folio *folio, *drained =LRU_CACHE_DRAINED_ALL ; } } +EXPORT_SYMBOL_FOR_KVM (lru_cache_drain_for_folio);atomic_t lru_disable_count =ATOMIC_INIT (0 );
I don't mind about the virt/kvm/guest_memfd.c part of it, but I'm finding a KVM patchset modifying mm/folio.c there hard to deal with: and notice Sean also suggesting to separate this part out.
As you know, I've worked up a patchset "mm/fbatch: drain lru_add_drain() and _all()" which finally removes the problem lru_cache_drain_for_folio() works around. In the initial version posted a week ago, there was no lru_cache_drain_for_folio() in the tree. Now 7.3-rc1 has it, so I intended a replacement 13/25 in my series, giving you just an empty inline lru_cache_drain_for_folio() stub (and enum lru_cache_drained) in linux/swap.h.
But that won't work for you, if you're adding an EXPORT_SYMBOL_FOR_KVM() in mm/folio.c, and of course conflicts with my removals (in context both above and below your EXPORT line). It's easy for me to remove what's in mm/gup.c and mm/folio.c, but I cannot remove what is not yet there.
I've wasted hours on this, hoping not to trouble either of you; but seeing now that I shall have to rebase anyway (an unrelated mlock fix), I'm electing to take the only clean way out: I'm going to submit this mm/folio.c part of your patch to Andrew tonight (with a shorter Cc list!), in the hope that it can be accelerated into 7.3-rc2 (or at least get an mm-stable stable base-commit id) which we can both work off independently.
Whether that's acceptable to Ackerley and to Andrew, I don't know (just as we don't know when either of our patchsets will go further), but let me try.
Thanks,
Hugh
Re: [PATCH v12 18/45] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
Quoted 29 lines · Hugh Dickins
[...snip...]
mm/folio.c void lru_cache_drain_for_folio(const struct folio *folio, · line 933
Quoted 8 lines
mm/folio.c +1
-0@@ -926,6 +927,7 @@ void lru_cache_drain_for_folio(const struct folio *folio, *drained =LRU_CACHE_DRAINED_ALL ; } } +EXPORT_SYMBOL_FOR_KVM (lru_cache_drain_for_folio);atomic_t lru_disable_count =ATOMIC_INIT (0 );I don't mind about the
virt/kvm/guest_memfd.cpart of it, but I'm finding a KVM patchset modifyingmm/folio.cthere hard to deal with: and notice Sean also suggesting to separate this part out.As you know, I've worked up a patchset "mm/fbatch: drain
lru_add_drain()and_all()" which finally removes the problemlru_cache_drain_for_folio()works around. In the initial version posted a week ago, there was nolru_cache_drain_for_folio()in the tree. Now 7.3-rc1 has it, so I intended a replacement 13/25 in my series, giving you just an empty inlinelru_cache_drain_for_folio()stub (and enumlru_cache_drained) inlinux/swap.h.But that won't work for you, if you're adding an
EXPORT_SYMBOL_FOR_KVM()inmm/folio.c, and of course conflicts with my removals (in context both above and below your EXPORT line). It's easy for me to remove what's inmm/gup.candmm/folio.c, but I cannot remove what is not yet there.
Quoted 7 lines
I've wasted hours on this, hoping not to trouble either of you; but seeing now that I shall have to rebase anyway (an unrelated mlock fix), I'm electing to take the only clean way out: I'm going to submit this
mm/folio.cpart of your patch to Andrew tonight (with a shorter Cc list!), in the hope that it can be accelerated into 7.3-rc2 (or at least get an mm-stable stable base-commit id) which we can both work off independently.
Thank you! Will use this in the next revision of this series.
Quoted 6 lines
Whether that's acceptable to Ackerley and to Andrew, I don't know (just as we don't know when either of our patchsets will go further), but let me try.
Thanks,
Hugh
Re: [PATCH v12 18/45] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
In reply to Ackerley TngQuoted 22 lines
virt/kvm/guest_memfd.c +3
-3+static bool kvm_gmem_has_outstanding_references (struct inode *inode,pgoff_t start,size_t nr_pages,pgoff_t *err_index) { +enum lru_cache_drained drained =LRU_CACHE_NOT_DRAINED ;struct address_space *mapping = inode->i_mapping;pgoff_t last = start + nr_pages -1 ;bool has_outstanding = false;@@ -570,17 +589,12 @@ static bool kvm_gmem_has_outstanding_references(struct inode *inode,folio_batch_init (&fbatch); next = start; -while (has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) { +while (!has_outstanding &&filemap_get_folios (mapping, &next, last, &fbatch)) {for (i =0 ; i <folio_batch_count (&fbatch); ++i) {struct folio *folio = fbatch.folios[i]; -/* -* Outstanding references are anything other than those
virt/kvm/guest_memfd.c static bool kvm_gmem_has_outstanding_references(struct inode *inode, · line 600
Quoted 10 lines
virt/kvm/guest_memfd.c +2
-5-* from the page cache, plus 1 temporary reference held -* by filemap_get_folios() in the folio batch. -*/ -if (folio_ref_count (folio) !=folio_nr_pages (folio) +1 ) { - has_outstanding = true; + has_outstanding =__folio_has_outstanding_references (folio, &drained); +if (has_outstanding) { *err_index =max (start, folio->index);break ; }
Having the folio.c bits go upstream early is certainly cleaner; without Hugh's series an ack from MM maintainers would have been sufficient, though.
In any case, on the revised patch with only guest_memfd changes:
Signature · 3 lines
Cheers, David
PATCH v12 19/45 KVM: guest_memfd: Zero page while getting pfn
In reply to Ackerley TngMove the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().
With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.
Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.
guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.
Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.
There may be some performance penalty due to redundant zeroing, but this would pale in comparison to the cost of actually assigning the page to the VM.
This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().
On populate failure, the page is not re-zeroed, since on SNP, if firmware rejects a CPUID page, the expected CPUID values provided by firmware are returned to userspace via page contents. More generally, page contents may be modified on populate failure.
Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.
--- virt/kvm/guest_memfd.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +5 -7
@@ -1105,6 +1105,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,return ERR_PTR (-EHWPOISON ); } +if (!folio_test_uptodate (folio)) { +clear_highpage (folio_page (folio,0 )); +folio_mark_uptodate (folio); + } + *pfn =folio_file_pfn (folio, index);if (max_order) *max_order =0 ;
@@ -1133,11 +1138,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, goto out; } -if (!folio_test_uptodate (folio)) { -clear_highpage (folio_page (folio,0 )); -folio_mark_uptodate (folio); - } -if (kvm_arch_has_gmem_convert () &&kvm_gmem_is_private_mem (file_inode (file), index)) r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
@@ -1179,8 +1179,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, } ret =post_populate (kvm, gfn, pfn, src_page, opaque); -if (!ret) -folio_mark_uptodate (folio); out_put_folio:folio_put (folio);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 19/45] KVM: guest_memfd: Zero page while getting pfn
Quoted 36 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Ackerley Tng <[email protected]>
Move the folio initialization logic from
kvm_gmem_get_pfn()into__kvm_gmem_get_pfn()to also zero pages if the page is to be used inkvm_gmem_populate().With in-place conversion, the existing data in a
guest_memfdpage can be populated into guest memory through platform-specific ioctls.Without first zeroing the page obtained using
__kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.
guest_memfdpages are zeroed at most once in the page's entire lifetime withguest_memfd, and that is tracked using the uptodate flag.Zeroing the page in
__kvm_gmem_get_pfn()is chosen over zeroing inkvm_gmem_get_folio()since other flows, such as a futurewrite()syscall, can get a page, write to the page and then set page uptodate without zeroing.There may be some performance penalty due to redundant zeroing, but this would pale in comparison to the cost of actually assigning the page to the VM.
This aligns with the concept of zeroing before first use - the other place where zeroing happens is in
kvm_gmem_fault_user_mapping().On populate failure, the page is not re-zeroed, since on SNP, if firmware rejects a CPUID page, the expected CPUID values provided by firmware are returned to userspace via page contents. More generally, page contents may be modified on populate failure.
Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the
post_populate()call.
Quoted 6 lines
Reviewed-by Fuad Tabba <[email protected]>Tested-by Shivank Garg <[email protected]>Reviewed-by Xiaoyao Li <[email protected]>Reviewed-by Binbin Wu <[email protected]>Signed-off-by Ackerley Tng <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v12 20/45 KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE
In reply to Ackerley TngFrom: Michael Roth <[email protected]>
Make the source page for populating an SNP guest_memfd instance optional if in-place conversion/population is enabled. If KVM can convert the page in-place, then it's possible for guest memory to be initialized directly from userspace by mmap()'ing the guest_memfd and writing to it while the corresponding GPA ranges are in a 'shared' state, before converting them to the 'private' state expected by KVM_SEV_SNP_LAUNCH_UPDATE.
Update the handling/documentation for KVM_SEV_SNP_LAUNCH_UPDATE to allow for 'uaddr' to be set to NULL when in-place conversion is enabled, which SNP_LAUNCH_UPDATE will then use to determine when it should/shouldn't copy in data from a separate memory location. Continue to enforce non-NULL when PRIVATE is tracked per-VM, not per-guest_memfd.
--- Documentation/virt/kvm/x86/amd-memory-encryption.rst | 14 ++++++++++---- arch/x86/kvm/svm/sev.c | 11 ++++++----- virt/kvm/kvm_main.c | 1 + 3 files changed, 17 insertions(+), 9 deletions(-)
Documentation/virt/kvm/x86/amd-memory-encryption.rst +10 -4
@@ -503,7 +503,8 @@ secrets. It is required that the GPA ranges initialized by this command have had the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute set in advance. See the documentation -for KVM_SET_MEMORY_ATTRIBUTES for more details on this aspect. +for KVM_SET_MEMORY_ATTRIBUTES/KVM_SET_MEMORY_ATTRIBUTES2 for more details on +this aspect. Upon success, this command is not guaranteed to have processed the entire range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of
@@ -511,9 +512,14 @@ range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of remaining range that has yet to be processed. The caller should continue calling this command until those fields indicate the entire range has been processed, e.g. ``len`` is 0, ``gfn_start`` is equal to the last GFN in the -range plus 1, and ``uaddr`` is the last byte of the userspace-provided source -buffer address plus 1. In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO, -``uaddr`` will be ignored completely. +range plus 1, and ``uaddr`` (if specified) is the last byte of the +userspace-provided source buffer address plus 1. + +In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO, ``uaddr`` will be +ignored completely. For all other page types, ``uaddr`` is optional if in-place +conversion is enabled (i.e. when the data had been written directly to +guest_memfd while the page was in the shared state) and is required if in-place +conversion is disabled. Parameters (in): struct kvm_sev_snp_launch_update
arch/x86/kvm/svm/sev.c +6 -5
@@ -2400,7 +2400,7 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,*/ if (ret && !snp_page_reclaim (kvm, pfn) && sev_populate_args->type ==KVM_SEV_SNP_PAGE_TYPE_CPUID && - sev_populate_args->fw_error ==SEV_RET_INVALID_PARAM ) { + sev_populate_args->fw_error ==SEV_RET_INVALID_PARAM && src_page) {void *src_vaddr =kmap_local_page (src_page);void *dst_vaddr =kmap_local_pfn (pfn);
@@ -2433,8 +2433,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)if (copy_from_user (¶ms,u64_to_user_ptr (argp->data),sizeof (params)))return -EFAULT ; -pr_debug ("%s: GFN start 0x%llx length 0x%llx type %d flags %d\n" , __func__, - params.gfn_start, params.len, params.type, params.flags); +pr_debug ("%s: GFN start 0x%llx length 0x%llx type %d flags %d src %llx\n" , __func__, + params.gfn_start, params.len, params.type, params.flags, params.uaddr); if (!params.len || !PAGE_ALIGNED(params.len) || params.flags || (params.type != KVM_SEV_SNP_PAGE_TYPE_NORMAL &&
@@ -2446,7 +2446,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)if (params.type ==KVM_SEV_SNP_PAGE_TYPE_ZERO ) src =NULL ; -else if (!params.uaddr || !PAGE_ALIGNED (params.uaddr)) +else if ((!gmem_in_place_conversion && !params.uaddr) || + !PAGE_ALIGNED (params.uaddr))return -EINVAL ;else src =u64_to_user_ptr (params.uaddr);
@@ -2493,7 +2494,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) params.gfn_start += count; params.len -= count *PAGE_SIZE ; -if (params.type !=KVM_SEV_SNP_PAGE_TYPE_ZERO ) +if (src && params.type !=KVM_SEV_SNP_PAGE_TYPE_ZERO ) params.uaddr += count *PAGE_SIZE ;if (copy_to_user (u64_to_user_ptr (argp->data), ¶ms,sizeof (params)))
virt/kvm/kvm_main.c +1 -0
@@ -104,6 +104,7 @@ module_param(allow_unsafe_mappings, bool, 0444);#ifdef kvm_arch_has_private_membool __ro_after_init gmem_in_place_conversion = false; +EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif /*
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 21/45 KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION
In reply to Ackerley TngUpdate tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.
Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.
Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.
Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.
--- Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++ arch/x86/kvm/vmx/tdx.c | 8 +++++--- 2 files changed, 9 insertions(+), 3 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +4 -0
@@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION Initialize @nr_pages TDX guest private memory starting from @gpa with userspace provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned. +If guest_memfd in-place conversion is enabled, pass 0 for @source_addr +to represent "no source page". A source page is required if in-place +conversion is not enabled or not supported. + Note, before calling this sub command, memory attribute of the range [gpa, gpa + nr_pages] needs to be private. Userspace can use KVM_SET_MEMORY_ATTRIBUTES to set the attribute.
arch/x86/kvm/vmx/tdx.c +5 -3
@@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,if (KVM_BUG_ON (kvm_tdx->page_add_src, kvm))return -EIO ; - kvm_tdx->page_add_src = src_page; + kvm_tdx->page_add_src = src_page ?:pfn_to_page (pfn); ret =kvm_tdp_mmu_map_private_pfn (arg->vcpu, gfn, pfn); kvm_tdx->page_add_src =NULL ;
@@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cif (copy_from_user (®ion,u64_to_user_ptr (cmd->data),sizeof (region)))return -EFAULT ; - if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr || + if (!PAGE_ALIGNED(region.source_addr) || + (!gmem_in_place_conversion && !region.source_addr) || !PAGE_ALIGNED(region.gpa) || !region.nr_pages || region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa || !vt_is_tdx_private_gpa(kvm, region.gpa) ||
@@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cbreak ; } - region.source_addr +=PAGE_SIZE ; +if (region.source_addr) + region.source_addr +=PAGE_SIZE ; region.gpa +=PAGE_SIZE ; region.nr_pages--;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 22/45 KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Bury KVM_VM_MEMORY_ATTRIBUTES in x86 to discourage other architectures from adding support for per-VM memory attributes, because tracking private vs. shared memory on a per-VM basis is now deprecated in favor of tracking on a per-guest_memfd basis, and while RWX memory attributes are on the horizon, they too are expected to be x86-only.
This will also allow modifying KVM_VM_MEMORY_ATTRIBUTES to be user-selectable (in x86) without creating weirdness in KVM's Kconfigs. Now that guest_memfd supports in-place conversions, it's entirely possible to run x86 CoCo VMs without support for KVM_VM_MEMORY_ATTRIBUTES.
Leave the code itself in common KVM so that it's trivial to undo this change if new per-VM attributes do come along.
--- arch/x86/kvm/Kconfig | 3 +++ virt/kvm/Kconfig | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/Kconfig +3 -0
@@ -80,6 +80,9 @@ config KVM_WERROR If in doubt, say "N". +config KVM_VM_MEMORY_ATTRIBUTES + bool + config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT
virt/kvm/Kconfig +0 -3
@@ -100,9 +100,6 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG config KVM_MMU_LOCKLESS_AGING bool -config KVM_VM_MEMORY_ATTRIBUTES - bool - config KVM_GUEST_MEMFD select XARRAY_MULTI bool
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 23/45 KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Allow the user to disable KVM_VM_MEMORY_ATTRIBUTES even when KVM supports PRIVATE and SHARED attributes, and expose gmem_in_place_conversion as a module parameter when per-VM attributes are supported. I.e. let userspace enable in-place PRIVATE<=>SHARED conversion of guest_memfd pages.
Provide both a Kconfig option and a (conditional) module param so that deployments that use a custom kernel can fully disable per-VM tracking, while not forcing distros to ship two separate kernels in order to provide backwards compatibility for downstream users.
Don't allow running VMs with mixed tracking for a given instance of KVM, i.e. disallow toggling the module param after KVM is loaded, as the extra complexity needed to handle per-VM behavior far outweighs any potential benefit. E.g. neither TDX nor SNP supports live migration, so in effect the requirement is that existing deployments that want to support both the old and the new models would need to tell their VMM which flavor of tracking to use.
--- Documentation/admin-guide/kernel-parameters.txt | 25 +++++++++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 4 +++- arch/x86/kvm/Kconfig | 14 ++++++++++---- virt/kvm/kvm_main.c | 5 ++++- 4 files changed, 42 insertions(+), 6 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +25 -0
@@ -3156,6 +3156,31 @@ Kernel parameters kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support). + kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place + conversion is unconditionally enabled. + + Default is Y (on). + kvm.nx_huge_pages= [KVM] Controls the software workaround for the X86_BUG_ITLB_MULTIHIT bug.
arch/x86/include/asm/kvm_host.h +3 -1
@@ -1851,7 +1851,9 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +#if defined(CONFIG_KVM_SW_PROTECTED_VM ) || \ + defined(CONFIG_KVM_INTEL_TDX ) || \ + defined(CONFIG_KVM_AMD_SEV )#define kvm_arch_has_private_mem (kvm) ((kvm)->arch.has_private_mem)#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
arch/x86/kvm/Kconfig +10 -4
@@ -81,13 +81,21 @@ config KVM_WERROR If in doubt, say "N". config KVM_VM_MEMORY_ATTRIBUTES - bool + bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)" + depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV + help + Enable support for tracking PRIVATE vs. SHARED memory using per-VM + memory attributes. Using per-VM attributes is deprecated in favor of + tracking PRIVATE state in guest_memfd. Select this if you need to run + CoCo VMs using a VMM that doesn't support guest_memfd memory + attributes. + + If unsure, say N. config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT depends on KVM_X86 && X86_64 - select KVM_VM_MEMORY_ATTRIBUTES help Enable support for KVM software-protected VMs. Currently, software- protected VMs are purely a development and testing vehicle for
@@ -138,7 +146,6 @@ config KVM_INTEL_TDX bool "Intel Trust Domain Extensions (TDX) support" default y depends on INTEL_TDX_HOST - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_POPULATE help Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -162,7 +169,6 @@ config KVM_AMD_SEV depends on KVM_AMD && X86_64 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE
virt/kvm/kvm_main.c +4 -1
@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...
Quoted 14 lines
Documentation/admin-guide/kernel-parameters.txt +14
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place + conversion is unconditionally enabled. + + Default is Y (on).
The parameter is only settable when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that config gmem_in_place_conversion initialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
Cheers,
/fuad
Quoted 84 lines
Documentation/admin-guide/kernel-parameters.txt +1
-0+ kvm.nx_huge_pages= [KVM] Controls the software workaround for the X86_BUG_ITLB_MULTIHIT bug.arch/x86/include/asm/kvm_host.h +3
-1@@ -1851,7 +1851,9 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +#if defined(CONFIG_KVM_SW_PROTECTED_VM ) || \ + defined(CONFIG_KVM_INTEL_TDX ) || \ + defined(CONFIG_KVM_AMD_SEV )#define kvm_arch_has_private_mem (kvm) ((kvm)->arch.has_private_mem)#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT arch/x86/kvm/Kconfig +10
-4@@ -81,13 +81,21 @@ config KVM_WERROR If in doubt, say "N". config KVM_VM_MEMORY_ATTRIBUTES - bool + bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)" + depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV + help + Enable support for tracking PRIVATE vs. SHARED memory using per-VM + memory attributes. Using per-VM attributes is deprecated in favor of + tracking PRIVATE state in guest_memfd. Select this if you need to run + CoCo VMs using a VMM that doesn't support guest_memfd memory + attributes. + + If unsure, say N. config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT depends on KVM_X86 && X86_64 - select KVM_VM_MEMORY_ATTRIBUTES help Enable support for KVM software-protected VMs. Currently, software- protected VMs are purely a development and testing vehicle for@@ -138,7 +146,6 @@ config KVM_INTEL_TDX bool "Intel Trust Domain Extensions (TDX) support" default y depends on INTEL_TDX_HOST - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_POPULATE help Provides support for launching Intel Trust Domain Extensions (TDX)@@ -162,7 +169,6 @@ config KVM_AMD_SEV depends on KVM_AMD && X86_64 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATEvirt/kvm/kvm_main.c +4
-2@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif --2.55.0.897.gb25b4bd76c-goog
Sean Christopherson
maintainerRe: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
Quoted 26 lines · Fuad Tabba
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...Quoted 21 lines
Documentation/admin-guide/kernel-parameters.txt +21
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ conversion is unconditionally enabled. + + Default is Y (on).The parameter is only settable when
CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that configgmem_in_place_conversioninitialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
The doc is wrong. The intent is to maintain backwards compatibility with existing deployments if KVM_VM_MEMORY_ATTRIBUTES=y.
Quoted 15 lines
Quoted 15 lines
virt/kvm/kvm_main.c +4
-1@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif
Re: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
Quoted 37 lines · Sean Christopherson
Quoted 26 lines · Fuad Tabba
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...Quoted 21 lines
Documentation/admin-guide/kernel-parameters.txt +21
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-placeQuoted 8 lines
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ conversion is unconditionally enabled. + + Default is Y (on).The parameter is only settable when
CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that configgmem_in_place_conversioninitialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
Quoted 2 lines
The doc is wrong. The intent is to maintain backwards compatibility with existing deployments if KVM_VM_MEMORY_ATTRIBUTES=y.
That explains it. Thanks.
/fuad
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
virt/kvm/kvm_main.c +4
-1@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif
Re: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
In reply to Sean ChristophersonQuoted 36 lines · Sean Christopherson
Quoted 26 lines · Fuad Tabba
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...Quoted 21 lines
Documentation/admin-guide/kernel-parameters.txt +21
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-placeQuoted 8 lines
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ conversion is unconditionally enabled. + + Default is Y (on).The parameter is only settable when
CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that configgmem_in_place_conversioninitialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
Quoted 3 lines
The doc is wrong. The intent is to maintain backwards compatibility with existing deployments if KVM_VM_MEMORY_ATTRIBUTES=y.
Will it result in the situation that kernel builds with CONFIG_KVM_VM_MEMORY_ATTRIBUTES enabled by default for backwards compatibility, which requires users to explicitly enable gmem_in_place_conversion by the kernel option? Then it effectively disables gmem_in_place_conversion by default.
But I guess it's OK for CSPs since they know what they want.
Quoted 16 lines
Quoted 15 lines
Quoted 15 lines
virt/kvm/kvm_main.c +4
-1@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif
Sean Christopherson
maintainerRe: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
Quoted 37 lines · Binbin Wu
Quoted 36 lines · Sean Christopherson
Quoted 34 lines · Fuad Tabba
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...Quoted 14 lines
Documentation/admin-guide/kernel-parameters.txt +14
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversionDocumentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place + conversion is unconditionally enabled. + + Default is Y (on).The parameter is only settable when
CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that configgmem_in_place_conversioninitialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
Quoted 8 lines
Quoted 3 lines
The doc is wrong. The intent is to maintain backwards compatibility with existing deployments if KVM_VM_MEMORY_ATTRIBUTES=y.
Will it result in the situation that kernel builds with
CONFIG_KVM_VM_MEMORY_ATTRIBUTESenabled by default for backwards compatibility, which requires users to explicitly enablegmem_in_place_conversionby the kernel option? Then it effectively disablesgmem_in_place_conversionby default.
Yes. My intent/expectation is that the only reason to build with CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y is for backwards compatibility, and so it makes sense to have the existing behavior be the default. If a deployment only wants the new behavior, then it can simply build with CONFIG_KVM_VM_MEMORY_ATTRIBUTES=n.
Folks that want only the new behavior but are using a distro/pre-built kernel will need to manually enable in-place conversion, but that's not too onerous, and it's simply not feasible to provide defaults that work for every possible scenario.
Quoted 1 line
But I guess it's OK for CSPs since they know what they want.
Yep.
Re: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
In reply to Sean ChristophersonQuoted 36 lines · Sean Christopherson
Quoted 26 lines · Fuad Tabba
Hi Sean, Ackerley,
On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay <[email protected]> wrote:
...Quoted 21 lines
Documentation/admin-guide/kernel-parameters.txt +21
-0+ kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-placeQuoted 8 lines
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ conversion is unconditionally enabled. + + Default is Y (on).The parameter is only settable when
CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y, and in that configgmem_in_place_conversioninitialises to !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES), i.e. N, not Y. Is the doc line wrong, or is N-by-default not what was intended here?
Quoted 3 lines
The doc is wrong. The intent is to maintain backwards compatibility with existing deployments if KVM_VM_MEMORY_ATTRIBUTES=y.
Thanks Fuad! I got confused, gmem_in_place_conversion the variable defaults to true (because CONFIG_KVM_VM_MEMORY_ATTRIBUTES is by default not set), but if the parameter exists, CONFIG_KVM_VM_MEMORY_ATTRIBUTES must be set and so the parameter defaults to N.
Will fix in the next revision.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
virt/kvm/kvm_main.c +4
-1@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 );#ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED (CONFIG_KVM_VM_MEMORY_ATTRIBUTES ); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param (gmem_in_place_conversion, bool,0444 ); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL (gmem_in_place_conversion);#endif
Re: [PATCH v12 23/45] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
In reply to Ackerley TngQuoted 57 lines · On 8/31/26 02:25, Ackerley Tng via B4 Relay wrote:
From: Sean Christopherson <[email protected]>
Allow the user to disable
KVM_VM_MEMORY_ATTRIBUTESeven when KVM supports PRIVATE and SHARED attributes, and exposegmem_in_place_conversionas a module parameter when per-VM attributes are supported. I.e. let userspace enable in-place PRIVATE<=>SHARED conversion ofguest_memfdpages.Provide both a
Kconfigoption and a (conditional) module param so that deployments that use a custom kernel can fully disable per-VM tracking, while not forcing distros to ship two separate kernels in order to provide backwards compatibility for downstream users.Don't allow running VMs with mixed tracking for a given instance of KVM, i.e. disallow toggling the module param after KVM is loaded, as the extra complexity needed to handle per-VM behavior far outweighs any potential benefit. E.g. neither TDX nor SNP supports live migration, so in effect the requirement is that existing deployments that want to support both the old and the new models would need to tell their VMM which flavor of tracking to use.
Signed-off-by Sean Christopherson <[email protected]>Reviewed-by Fuad Tabba <[email protected]>Tested-by Shivank Garg <[email protected]>[Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled]Suggested-by Xiaoyao Li <[email protected]>Reviewed-by Xiaoyao Li <[email protected]>Co-developed-by Ackerley Tng <[email protected]>Signed-off-by Ackerley Tng <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 25 +++++++++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 4 +++- arch/x86/kvm/Kconfig | 14 ++++++++++---- virt/kvm/kvm_main.c | 5 ++++- 4 files changed, 42 insertions(+), 6 deletions(-)Documentation/admin-guide/kernel-parameters.txt +14
-0@@ -3156,6 +3156,31 @@ Kernel parameters kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support). + kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 3182
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place + conversion is unconditionally enabled. + + Default is Y (on).
With the doc fixed
Signature · 3 lines
Cheers, David
PATCH v12 24/45 KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Now that guest_memfd supports tracking private vs. shared within gmem itself, allow userspace to specify INIT_SHARED on a guest_memfd instance for x86 Confidential Computing (CoCo) VMs, so long as in-place conversion is enabled, i.e. when it's actually possible for a guest_memfd instance to contain shared memory.
--- arch/x86/kvm/x86.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/kvm/x86.c +7 -6
@@ -10637,14 +10637,15 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu) }#ifdef CONFIG_KVM_GUEST_MEMFD -/* -* KVM doesn't yet support initializing guest_memfd memory as shared for VMs -* with private memory (the private vs. shared tracking needs to be moved into -* guest_memfd). -*/ bool kvm_arch_supports_gmem_init_shared (struct kvm *kvm) { -return !kvm_arch_has_private_mem (kvm); +/* +* INIT_SHARED is supported if in-place conversion is enabled, or if +* the VM doesn't support private memory. If the VM has private memory +* and in-place conversion is disabled, then guest_memfd can _only_ be +* used for private memory. +*/ +return gmem_in_place_conversion || !kvm_arch_has_private_mem (kvm); }#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 25/45 KVM: selftests: Create gmem fd before "regular" fd when adding memslot
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
When adding a memslot associated with a guest_memfd instance, create/dup the guest_memfd before creating the "normal" backing file. This will allow dup'ing the gmem fd as the normal fd when guest_memfd supports mmap(), i.e. to make guest_memfd the _only_ backing source for the memslot.
--- tools/testing/selftests/kvm/lib/kvm_util.c | 45 +++++++++++++++--------------- 1 file changed, 23 insertions(+), 22 deletions(-)
tools/testing/selftests/kvm/lib/kvm_util.c +23 -22
@@ -1077,6 +1077,29 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,if (alignment >1 ) region->mmap_size += alignment; +if (flags &KVM_MEM_GUEST_MEMFD ) { +if (guest_memfd <0 ) { +u32 guest_memfd_flags =0 ; + +TEST_ASSERT (!guest_memfd_offset, +"Offset must be zero when creating new guest_memfd" ); + guest_memfd =vm_create_guest_memfd (vm, mem_size, guest_memfd_flags); + }else { +/* +* Install a unique fd for each memslot so that the fd +* can be closed when the region is deleted without +* needing to track if the fd is owned by the framework +* or by the caller. +*/ + guest_memfd =kvm_dup (guest_memfd); + } + + region->region.guest_memfd = guest_memfd; + region->region.guest_memfd_offset = guest_memfd_offset; + }else { + region->region.guest_memfd =-1 ; + } + region->fd =-1 ;if (backing_src_is_shared (src_type)) region->fd = kvm_memfd_alloc(region->mmap_size,
@@ -1106,28 +1129,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->backing_src_type = src_type; -if (flags &KVM_MEM_GUEST_MEMFD ) { -if (guest_memfd <0 ) { -u32 guest_memfd_flags =0 ; -TEST_ASSERT (!guest_memfd_offset, -"Offset must be zero when creating new guest_memfd" ); - guest_memfd =vm_create_guest_memfd (vm, mem_size, guest_memfd_flags); - }else { -/* -* Install a unique fd for each memslot so that the fd -* can be closed when the region is deleted without -* needing to track if the fd is owned by the framework -* or by the caller. -*/ - guest_memfd =kvm_dup (guest_memfd); - } - - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; - }else { - region->region.guest_memfd =-1 ; - } - region->unused_phy_pages =sparsebit_alloc ();if (vm_arch_has_protected_memory (vm)) region->protected_phy_pages =sparsebit_alloc ();
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 26/45 KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Rename local variables and function parameters for the guest memory file descriptor and its offset to use a "gmem_" prefix instead of "guest_memfd_".
No functional change intended.
--- tools/testing/selftests/kvm/include/kvm_util.h | 6 +++--- tools/testing/selftests/kvm/lib/kvm_util.c | 26 +++++++++++++------------- 2 files changed, 16 insertions(+), 16 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -3
@@ -690,17 +690,17 @@ int __vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva);void vm_set_user_memory_region2 (struct kvm_vm *vm,u32 slot,u32 flags,gpa_t gpa,u64 size,void *hva, -u32 guest_memfd,u64 guest_memfd_offset); +u32 gmem_fd,u64 gmem_offset);int __vm_set_user_memory_region2 (struct kvm_vm *vm,u32 slot,u32 flags,gpa_t gpa,u64 size,void *hva, -u32 guest_memfd,u64 guest_memfd_offset); +u32 gmem_fd,u64 gmem_offset);void vm_userspace_mem_region_add (struct kvm_vm *vm,enum vm_mem_backing_src_type src_type,gpa_t gpa,u32 slot,u64 npages,u32 flags);void vm_mem_add (struct kvm_vm *vm,enum vm_mem_backing_src_type src_type,gpa_t gpa,u32 slot,u64 npages,u32 flags, -int guest_memfd_fd,u64 guest_memfd_offset); +int gmem_fd,u64 gmem_offset);#ifndef vm_arch_has_protected_memorystatic inline bool vm_arch_has_protected_memory (struct kvm_vm *vm)
tools/testing/selftests/kvm/lib/kvm_util.c +13 -13
@@ -970,7 +970,7 @@ void vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags,int __vm_set_user_memory_region2 (struct kvm_vm *vm,u32 slot,u32 flags,gpa_t gpa,u64 size,void *hva, -u32 guest_memfd,u64 guest_memfd_offset) +u32 gmem_fd,u64 gmem_offset) {struct kvm_userspace_memory_region2 region = { .slot = slot,
@@ -978,8 +978,8 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, .guest_phys_addr = gpa, .memory_size = size, .userspace_addr = (uintptr_t)hva, - .guest_memfd = guest_memfd, - .guest_memfd_offset = guest_memfd_offset, + .guest_memfd = gmem_fd, + .guest_memfd_offset = gmem_offset, };TEST_REQUIRE_SET_USER_MEMORY_REGION2 ();
@@ -989,10 +989,10 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,void vm_set_user_memory_region2 (struct kvm_vm *vm,u32 slot,u32 flags,gpa_t gpa,u64 size,void *hva, -u32 guest_memfd,u64 guest_memfd_offset) +u32 gmem_fd,u64 gmem_offset) {int ret =__vm_set_user_memory_region2 (vm, slot, flags, gpa, size, hva, - guest_memfd, guest_memfd_offset); + gmem_fd, gmem_offset);TEST_ASSERT (!ret,"KVM_SET_USER_MEMORY_REGION2 failed, errno = %d (%s)" , errno,strerror (errno));
@@ -1002,7 +1002,7 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,/* FIXME: This thing needs to be ripped apart and rewritten. */ void vm_mem_add (struct kvm_vm *vm,enum vm_mem_backing_src_type src_type,gpa_t gpa,u32 slot,u64 npages,u32 flags, -int guest_memfd,u64 guest_memfd_offset) +int gmem_fd,u64 gmem_offset) {int ret;struct userspace_mem_region *region;
@@ -1078,12 +1078,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->mmap_size += alignment;if (flags &KVM_MEM_GUEST_MEMFD ) { -if (guest_memfd <0 ) { -u32 guest_memfd_flags =0 ; +if (gmem_fd <0 ) { +u32 gmem_flags =0 ; -TEST_ASSERT (!guest_memfd_offset, +TEST_ASSERT (!gmem_offset,"Offset must be zero when creating new guest_memfd" ); - guest_memfd =vm_create_guest_memfd (vm, mem_size, guest_memfd_flags); + gmem_fd =vm_create_guest_memfd (vm, mem_size, gmem_flags); }else {/* * Install a unique fd for each memslot so that the fd
@@ -1091,11 +1091,11 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,* needing to track if the fd is owned by the framework * or by the caller. */ - guest_memfd =kvm_dup (guest_memfd); + gmem_fd =kvm_dup (gmem_fd); } - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; + region->region.guest_memfd = gmem_fd; + region->region.guest_memfd_offset = gmem_offset; }else { region->region.guest_memfd =-1 ; }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 27/45 KVM: selftests: Add support for mmap() on guest_memfd in core library
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Accept gmem_flags in vm_mem_add() to be able to create a guest_memfd within vm_mem_add().
When vm_mem_add() is used to set up a guest_memfd for a memslot, set up the provided (or created) gmem_fd as the fd for the user memory region. This makes it available to be mmap()-ed from just like fds from other memory sources.
For guest_memfds, mmap() using gmem_offset instead of 0 all the time.
Always use MAP_SHARED if mmap-ing from guest_memfd instead of reading flag from the configured src_type, which doesn't include guest_memfd.
Add a kvm_slot_to_fd() helper to provide convenient access to the file descriptor of a memslot.
Update existing callers of vm_mem_add() to pass 0 for gmem_flags to preserve existing behavior.
--- tools/testing/selftests/kvm/include/kvm_util.h | 7 +++++- tools/testing/selftests/kvm/lib/kvm_util.c | 28 ++++++++++++---------- .../kvm/x86/private_mem_conversions_test.c | 2 +- 3 files changed, 23 insertions(+), 14 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +6 -1
@@ -700,7 +700,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, gpa_t gpa, u32 slot, u64 npages, u32 flags);void vm_mem_add (struct kvm_vm *vm,enum vm_mem_backing_src_type src_type,gpa_t gpa,u32 slot,u64 npages,u32 flags, -int gmem_fd,u64 gmem_offset); +int gmem_fd,u64 gmem_offset,u64 gmem_flags);#ifndef vm_arch_has_protected_memorystatic inline bool vm_arch_has_protected_memory (struct kvm_vm *vm)
@@ -732,6 +732,11 @@ void *addr_gva2hva(struct kvm_vm *vm, gva_t gva);gpa_t addr_hva2gpa (struct kvm_vm *vm,void *hva);void *addr_gpa2alias (struct kvm_vm *vm,gpa_t gpa); +static inline int kvm_slot_to_fd (struct kvm_vm *vm,u32 slot) +{ +return memslot2region (vm, slot)->fd; +} +#ifndef vcpu_arch_put_guest#define vcpu_arch_put_guest (mem, val) do { (mem) = (val); } while (0)#endif
tools/testing/selftests/kvm/lib/kvm_util.c +16 -12
@@ -1002,12 +1002,14 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,/* FIXME: This thing needs to be ripped apart and rewritten. */ void vm_mem_add (struct kvm_vm *vm,enum vm_mem_backing_src_type src_type,gpa_t gpa,u32 slot,u64 npages,u32 flags, -int gmem_fd,u64 gmem_offset) +int gmem_fd,u64 gmem_offset,u64 gmem_flags) {int ret;struct userspace_mem_region *region;size_t backing_src_pagesz =get_backing_src_pagesz (src_type); +int mmap_flags =vm_mem_backing_src_alias (src_type)->flag;size_t mem_size = npages * vm->page_size; +off_t mmap_offset =0 ;size_t alignment =1 ;TEST_REQUIRE_SET_USER_MEMORY_REGION2 ();
@@ -1079,8 +1081,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,if (flags &KVM_MEM_GUEST_MEMFD ) {if (gmem_fd <0 ) { -u32 gmem_flags =0 ; -TEST_ASSERT (!gmem_offset,"Offset must be zero when creating new guest_memfd" ); gmem_fd =vm_create_guest_memfd (vm, mem_size, gmem_flags);
@@ -1101,13 +1101,17 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, } region->fd =-1 ; -if (backing_src_is_shared (src_type)) +if (flags &KVM_MEM_GUEST_MEMFD && gmem_flags &GUEST_MEMFD_FLAG_MMAP ) { + region->fd =kvm_dup (gmem_fd); + mmap_flags =MAP_SHARED ; + mmap_offset = gmem_offset; + }else if (backing_src_is_shared (src_type)) { region->fd =kvm_memfd_alloc (region->mmap_size, src_type ==VM_MEM_SRC_SHARED_HUGETLB ); + } - region->mmap_start =kvm_mmap (region->mmap_size,PROT_READ |PROT_WRITE , -vm_mem_backing_src_alias (src_type)->flag, - region->fd); + region->mmap_start =__kvm_mmap (region->mmap_size,PROT_READ |PROT_WRITE , + mmap_flags, region->fd, mmap_offset); TEST_ASSERT(!is_backing_src_hugetlb(src_type) || region->mmap_start == align_ptr_up(region->mmap_start, backing_src_pagesz),
@@ -1153,10 +1157,10 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,/* If shared memory, create an alias. */ if (region->fd >=0 ) { - region->mmap_alias =kvm_mmap (region->mmap_size, -PROT_READ |PROT_WRITE , -vm_mem_backing_src_alias (src_type)->flag, - region->fd); + region->mmap_alias =__kvm_mmap (region->mmap_size, +PROT_READ |PROT_WRITE , + mmap_flags, region->fd, + mmap_offset);/* Align host alias address */ region->host_alias =align_ptr_up (region->mmap_alias, alignment);
@@ -1167,7 +1171,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags) { - vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0); + vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0, 0); }/*
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -1
@@ -399,7 +399,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_vfor (i =0 ; i < nr_memslots; i++)vm_mem_add (vm, src_type,BASE_DATA_GPA + slot_size * i,BASE_DATA_SLOT + i, slot_size / vm->page_size, -KVM_MEM_GUEST_MEMFD , memfd, slot_size * i); +KVM_MEM_GUEST_MEMFD , memfd, slot_size * i,0 );for (i =0 ; i < nr_vcpus; i++) {gpa_t gpa =BASE_DATA_GPA + i * per_cpu_size;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 28/45 KVM: selftests: Add selftests global for guest memory attributes capability
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Add a global variable, kvm_has_gmem_attributes, to make the result of checking for KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES available to all tests.
kvm_has_gmem_attributes is true if guest_memfd tracks memory attributes, as opposed to VM-level tracking.
This global variable is synced to the guest for testing convenience, to avoid introducing subtle bugs when host/guest state is desynced.
--- tools/testing/selftests/kvm/include/test_util.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 5 +++++ 2 files changed, 7 insertions(+)
tools/testing/selftests/kvm/include/test_util.h +2 -0
@@ -116,6 +116,8 @@ struct kvm_random_state {extern u32 kvm_random_seed;extern struct kvm_random_state kvm_rng; +extern bool kvm_has_gmem_attributes; +struct kvm_random_state new_kvm_random_state (u32 seed);u32 kvm_random_u32 (struct kvm_random_state *state);
tools/testing/selftests/kvm/lib/kvm_util.c +5 -0
@@ -32,6 +32,8 @@ static void kvm_seed_rng(u32 seed) kvm_rng =new_kvm_random_state (kvm_random_seed); } +bool kvm_has_gmem_attributes; +static size_t vcpu_mmap_sz (void );int __open_path_or_exit (const char *path,int flags,const char *enoent_help)
@@ -527,6 +529,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus,kvm_seed_rng (kvm_random_seed);sync_global_to_guest (vm, kvm_rng); +sync_global_to_guest (vm, kvm_has_gmem_attributes);kvm_arch_vm_post_create (vm, nr_runnable_vcpus);
@@ -2311,6 +2314,8 @@ void __attribute((constructor)) kvm_selftest_init(void)srandom (time (0 ));kvm_seed_rng (random ()); + kvm_has_gmem_attributes =kvm_has_cap (KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES ); +kvm_selftest_arch_init (); }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 29/45 KVM: selftests: Add helpers for calling ioctls on guest_memfd
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Add helper functions to kvm_util.h to support calling ioctls, specifically KVM_SET_MEMORY_ATTRIBUTES2, on a guest_memfd file descriptor.
Introduce gmem_ioctl() and __gmem_ioctl() macros, modeled after the existing vm_ioctl() helpers, to provide a standard way to call ioctls on a guest_memfd.
Add gmem_set_memory_attributes() and its derivatives (gmem_set_private(), gmem_set_shared()) to set memory attributes on a guest_memfd region. Also provide "__" variants that return the ioctl error code instead of aborting the test. These helpers will be used by upcoming guest_memfd tests.
To avoid code duplication, factor out the check for supported memory attributes into a new macro, TEST_ASSERT_SUPPORTED_ATTRIBUTES, and use it in both the existing vm_set_memory_attributes() and the new gmem_set_memory_attributes() helpers.
--- tools/testing/selftests/kvm/include/kvm_util.h | 94 +++++++++++++++++++++++--- 1 file changed, 86 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +86 -8
@@ -392,6 +392,16 @@ static __always_inline void static_assert_is_vcpu(struct kvm_vcpu *vcpu) { } __TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (vcpu)->vm); \ }) +#define __gmem_ioctl (gmem_fd, cmd, arg) \ + kvm_do_ioctl(gmem_fd, cmd, arg) + +#define gmem_ioctl (gmem_fd, cmd, arg) \ +({ \ + int ret = __gmem_ioctl(gmem_fd, cmd, arg); \ + \ + TEST_ASSERT(!ret, __KVM_IOCTL_ERROR(#cmd, ret)); \ +}) +/* * Looks up and returns the value corresponding to the capability * (KVM_CAP_*) given by cap.
@@ -418,8 +428,16 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)vm_ioctl (vm,KVM_ENABLE_CAP , &enable_cap); } +/* +* KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes. These +* flows need significant enhancements to support multiple attributes. +*/ +#define TEST_ASSERT_SUPPORTED_ATTRIBUTES (attributes) \ + TEST_ASSERT(!(attributes) || (attributes) == KVM_MEMORY_ATTRIBUTE_PRIVATE, \ + "Update me to support multiple attributes!") +static inline void vm_set_memory_attributes (struct kvm_vm *vm,gpa_t gpa, -u64 size,u64 attributes) +size_t size,u64 attributes) {struct kvm_memory_attributes attr = { .attributes = attributes,
@@ -428,17 +446,11 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa, .flags = 0, }; -/* -* KVM_SET_MEMORY_ATTRIBUTES overwrites _all_ attributes. These flows -* need significant enhancements to support multiple attributes. -*/ -TEST_ASSERT (!attributes || attributes ==KVM_MEMORY_ATTRIBUTE_PRIVATE , -"Update me to support multiple attributes!" ); +TEST_ASSERT_SUPPORTED_ATTRIBUTES (attributes);vm_ioctl (vm,KVM_SET_MEMORY_ATTRIBUTES , &attr); } -static inline void vm_mem_set_private (struct kvm_vm *vm,gpa_t gpa,u64 size) {
@@ -451,6 +463,72 @@ static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa,vm_set_memory_attributes (vm, gpa, size,0 ); } +static inline int __gmem_set_memory_attributes (int fd,u64 offset, +size_t size,u64 attributes, +u64 *error_offset) +{ +struct kvm_memory_attributes2 attr = { + .attributes = attributes, + .offset = offset, + .size = size, + .flags =0 , + .error_offset =0 , + }; +int r; + + r =__gmem_ioctl (fd,KVM_SET_MEMORY_ATTRIBUTES2 , &attr); + +/* Copy error_offset regardless of r so caller can check. */ +if (error_offset) + *error_offset = attr.error_offset; + +return r; +} + +static inline int __gmem_set_private (int fd,u64 offset,size_t size, +u64 *error_offset) +{ +return __gmem_set_memory_attributes (fd, offset, size, +KVM_MEMORY_ATTRIBUTE_PRIVATE , + error_offset); +} + +static inline int __gmem_set_shared (int fd,u64 offset,size_t size, +u64 *error_offset) +{ +return __gmem_set_memory_attributes (fd, offset, size,0 , + error_offset); +} + +static inline void gmem_set_memory_attributes (int fd,u64 offset, +size_t size,u64 attributes) +{ +struct kvm_memory_attributes2 attr = { + .attributes = attributes, + .offset = offset, + .size = size, + .flags =0 , + }; + +TEST_ASSERT_SUPPORTED_ATTRIBUTES (attributes); + +__TEST_REQUIRE (kvm_check_cap (KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES ) >0 , +"No valid attributes for guest_memfd ioctl!" ); + +gmem_ioctl (fd,KVM_SET_MEMORY_ATTRIBUTES2 , &attr); +} + +static inline void gmem_set_private (int fd,u64 offset,size_t size) +{ +gmem_set_memory_attributes (fd, offset, size, +KVM_MEMORY_ATTRIBUTE_PRIVATE ); +} + +static inline void gmem_set_shared (int fd,u64 offset,size_t size) +{ +gmem_set_memory_attributes (fd, offset, size,0 ); +} +void vm_guest_mem_fallocate (struct kvm_vm *vm,gpa_t gpa,u64 size,bool punch_hole);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 30/45 KVM: selftests: Test basic single-page conversion flow
In reply to Ackerley TngAdd a selftest for the guest_memfd memory attribute conversion ioctls. The test starts the guest_memfd as all-private (the default state), and verifies the basic flow of converting a single page to shared and then back to private.
Add infrastructure that supports extensions to other conversion flow tests. This infrastructure will be used in upcoming patches for other conversion tests.
Add test as an x86-specific test since guest_memfd's testing vehicle (KVM_X86_SW_PROTECTED_VM) is x86-specific.
---
v10: Updated __GMEM_CONVERSION_TEST macro to avoid potential macro argument
reuse
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/x86/guest_memfd_conversions_test.c | 200 +++++++++++++++++++++ 2 files changed, 201 insertions(+)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -155,6 +155,7 @@ TEST_GEN_PROGS_x86 += x86/max_vcpuid_cap_test TEST_GEN_PROGS_x86 += x86/triple_fault_event_test TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test TEST_GEN_PROGS_x86 += x86/aperfmperf_test +TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test TEST_GEN_PROGS_x86 += access_tracking_perf_test TEST_GEN_PROGS_x86 += coalesced_io_test TEST_GEN_PROGS_x86 += dirty_log_perf_test
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 31/45 KVM: selftests: Test conversion flow when INIT_SHARED
In reply to Ackerley TngAdd a test case to verify that conversions between private and shared memory work correctly when the memory is initially created as shared.
--- .../selftests/kvm/x86/guest_memfd_conversions_test.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +13 -0
@@ -97,6 +97,12 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages) \#define GMEM_CONVERSION_TEST_INIT_PRIVATE (test) \ __GMEM_CONVERSION_TEST_INIT_PRIVATE(test, 1) +#define __GMEM_CONVERSION_TEST_INIT_SHARED (test, __nr_pages) \ + GMEM_CONVERSION_TEST(test, __nr_pages, GUEST_MEMFD_FLAG_INIT_SHARED) + +#define GMEM_CONVERSION_TEST_INIT_SHARED (test) \ + __GMEM_CONVERSION_TEST_INIT_SHARED(test, 1) +struct guest_check_data {void *mem;char expected_val;
@@ -188,6 +194,13 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(init_private)test_convert_to_private (t,0 ,'C' ,'E' ); } +GMEM_CONVERSION_TEST_INIT_SHARED (init_shared) +{ +test_shared (t,0 ,0 ,'A' ,'B' ); +test_convert_to_private (t,0 ,'B' ,'C' ); +test_convert_to_shared (t,0 ,'C' ,'D' ,'E' ); +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 32/45 KVM: selftests: Test conversion precision in guest_memfd
In reply to Ackerley TngThe existing guest_memfd conversion tests only use single-page memory regions. This provides no coverage for multi-page guest_memfd objects, specifically whether KVM correctly handles the page index for conversion operations. An incorrect implementation could, for example, always operate on the first page regardless of the index provided.
Add a new test case to verify that conversions between private and shared memory correctly target the specified page within a multi-page guest_memfd.
This test also verifies the precision of memory conversions by converting a single page and then iterating through all other pages to ensure they remain in their original state.
To support this test, add a new GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED macro that handles setting up and tearing down the VM for each page iteration. The teardown logic is adjusted to prevent a double-free in this new scenario.
---
v10: Update macro GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED to avoid
potential macro argument reuse
--- .../kvm/x86/guest_memfd_conversions_test.c | 67 ++++++++++++++++++++++ 1 file changed, 67 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +67 -0
@@ -61,8 +61,13 @@ static void gmem_conversions_do_setup(test_data_t *t, int nr_pages,static void gmem_conversions_do_teardown (test_data_t *t) { +/* Use NULL to avoid second free in FIXTURE_TEARDOWN (multipage tests). */ +if (!t->vcpu) +return ; +/* No need to close gmem_fd, it's owned by the VM structure. */ kvm_vm_free (t->vcpu->vm); + t->vcpu =NULL ; }FIXTURE_TEARDOWN (gmem_conversions)
@@ -103,6 +108,30 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages) \#define GMEM_CONVERSION_TEST_INIT_SHARED (test) \ __GMEM_CONVERSION_TEST_INIT_SHARED(test, 1) +/* +* Repeats test over nr_pages in a guest_memfd of size nr_pages, providing each +* test iteration with test_page, the index of the page under test in +* guest_memfd. test_page takes values 0..(nr_pages - 1) inclusive. +*/ +#define GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED (test, __nr_pages) \ +static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages, \ + const int test_page); \ + \ +TEST_F(gmem_conversions, test) \ +{ \ + const u64 flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED; \ + const int nr = (__nr_pages); \ + int i; \ + \ + for (i = 0; i < nr; ++i) { \ + gmem_conversions_do_setup(self, nr, flags); \ + __gmem_conversions_multipage_##test(self, nr, i); \ + gmem_conversions_do_teardown(self); \ + } \ +} \ +static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages, \ + const int test_page) +struct guest_check_data {void *mem;char expected_val;
@@ -201,6 +230,44 @@ GMEM_CONVERSION_TEST_INIT_SHARED(init_shared)test_convert_to_shared (t,0 ,'C' ,'D' ,'E' ); } +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4) +{ +int i; + +/* Get a char that varies with both i and n. */ +#define combine (x, n) (((x) << 4) + (n)) +#define i_ (n) (combine(i, n)) +#define t_ (n) (combine(test_page, n)) + +/* +* Start with the highest index, to catch any errors when, perhaps, the +* first page is returned even for the last index. +*/ +for (i = nr_pages -1 ; i >=0 ; --i) +test_shared (t, i,0 ,i_ (0 ),i_ (2 )); + +test_convert_to_private (t, test_page,t_ (2 ),t_ (3 )); + +for (i =0 ; i < nr_pages; ++i) { +if (i == test_page) +test_private (t, test_page,t_ (3 ),t_ (4 )); +else +test_shared (t, i,i_ (2 ),i_ (3 ),i_ (4 )); + } + +test_convert_to_shared (t, test_page,t_ (4 ),t_ (5 ),t_ (6 )); + +for (i =0 ; i < nr_pages; ++i) { +char expected = i == test_page ?t_ (6 ) :i_ (4 ); + +test_shared (t, i, expected,i_ (7 ),i_ (8 )); + } + +#undef t_ +#undef i_ +#undef combine +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 33/45 KVM: selftests: Test conversion before allocation
In reply to Ackerley TngAdd two test cases to the guest_memfd conversions selftest to cover the scenario where a conversion is requested before any memory has been allocated in the guest_memfd region.
The KVM_SET_MEMORY_ATTRIBUTES2 ioctl can be called on a memory region at any time. If the guest had not yet faulted in any pages for that region, the kernel must record the conversion request and apply the requested state when the pages are eventually allocated.
The new tests cover both conversion directions.
--- .../selftests/kvm/x86/guest_memfd_conversions_test.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +14 -0
@@ -268,6 +268,20 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4)#undef combine } +/* +* Test that even if there are no folios yet, conversion requests are recorded +* in guest_memfd. +*/ +GMEM_CONVERSION_TEST_INIT_SHARED (before_allocation_shared) +{ +test_convert_to_private (t,0 ,0 ,'A' ); +} + +GMEM_CONVERSION_TEST_INIT_PRIVATE (before_allocation_private) +{ +test_convert_to_shared (t,0 ,0 ,'A' ,'B' ); +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 34/45 KVM: selftests: Convert with allocated folios in different layouts
In reply to Ackerley TngAdd a guest_memfd selftest to verify that memory conversions work correctly with allocated folios in different layouts.
By iterating through which pages are initially faulted, the test covers various layouts of contiguous allocated and unallocated regions, exercising conversion with different range layouts.
--- .../kvm/x86/guest_memfd_conversions_test.c | 30 ++++++++++++++++++++++ 1 file changed, 30 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +30 -0
@@ -282,6 +282,36 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(before_allocation_private)test_convert_to_shared (t,0 ,0 ,'A' ,'B' ); } +/* +* Test that when some of the folios in the conversion range are allocated, +* conversion requests are handled correctly in guest_memfd. Vary the ranges +* allocated before conversion, using test_page, to cover various layouts of +* contiguous allocated and unallocated regions. +*/ +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8) +{ +const int second_page_to_fault =4 ; +int i; + +/* +* Fault 2 of the pages to test filemap range operations except when +* test_page == second_page_to_fault. +*/ +host_do_rmw (t->mem, test_page,0 ,'A' ); +if (test_page != second_page_to_fault) +host_do_rmw (t->mem, second_page_to_fault,0 ,'A' ); + +gmem_set_private (t->gmem_fd,0 , nr_pages * page_size); +for (i =0 ; i < nr_pages; ++i) { +char expected = (i == test_page || i == second_page_to_fault) ?'A' :0 ; + +test_private (t, i, expected,'B' ); + } + +for (i =0 ; i < nr_pages; ++i) +test_convert_to_shared (t, i,'B' ,'C' ,'D' ); +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 35/45 KVM: selftests: Test that truncation does not change shared/private status
In reply to Ackerley TngAdd a test to verify that deallocating a page in a guest memfd region via fallocate() with FALLOC_FL_PUNCH_HOLE does not alter the shared or private status of the corresponding memory range.
When a page backing a guest memfd mapping is deallocated, e.g., by punching a hole or truncating the file, and then subsequently faulted back in, the new page must inherit the correct shared/private status tracked by guest_memfd.
--- .../selftests/kvm/x86/guest_memfd_conversions_test.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +14 -0
@@ -10,6 +10,7 @@#include <linux/sizes.h> #include "kvm_util.h" +#include "kvm_syscalls.h" #include "kselftest_harness.h" #include "test_util.h" #include "ucall_common.h"
@@ -312,6 +313,19 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8)test_convert_to_shared (t, i,'B' ,'C' ,'D' ); } +/* Truncation should not affect shared/private status. */ +GMEM_CONVERSION_TEST_INIT_SHARED (truncate) +{ +host_do_rmw (t->mem,0 ,0 ,'A' ); +kvm_fallocate (t->gmem_fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE ,0 , page_size); +host_do_rmw (t->mem,0 ,0 ,'A' ); + +test_convert_to_private (t,0 ,'A' ,'B' ); + +kvm_fallocate (t->gmem_fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE ,0 , page_size); +test_private (t,0 ,0 ,'A' ); +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 36/45 KVM: selftests: Test that shared/private status is consistent across processes
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Add a test to verify that a guest_memfd's shared/private status is consistent across processes, and that any shared pages previously mapped in any process are unmapped from all processes.
The test forks a child process after creating the shared guest_memfd region so that the second process exists alongside the main process for the entire test.
The processes then take turns to access memory to check that the shared/private status is consistent across processes.
--- .../kvm/x86/guest_memfd_conversions_test.c | 118 +++++++++++++++++++++ 1 file changed, 118 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +118 -0
@@ -2,7 +2,9 @@/* * Copyright (c) 2024, Google LLC. */ +#include <pthread.h> #include <sys/mman.h> +#include <time.h> #include <unistd.h> #include <linux/align.h>
@@ -326,6 +328,122 @@ GMEM_CONVERSION_TEST_INIT_SHARED(truncate)test_private (t,0 ,0 ,'A' ); } +/* Test that shared/private memory protections work and are seen from any process. */ +GMEM_CONVERSION_TEST_INIT_SHARED (forked_accesses) +{ +enum test_state { +STATE_INIT , +STATE_CHECK_SHARED , +STATE_DONE_CHECKING_SHARED , +STATE_CHECK_PRIVATE , +STATE_DONE_CHECKING_PRIVATE , + }; + +struct sync_state { +pthread_mutex_t mutex; +pthread_cond_t cond; +enum test_state step; + } *sync; + +pthread_mutexattr_t mattr; +pthread_condattr_t cattr; +pid_t child_pid, parent_pid; +int status; + + sync =kvm_mmap (sizeof (*sync),PROT_READ |PROT_WRITE , +MAP_SHARED |MAP_ANONYMOUS ,-1 ); + +pthread_mutexattr_init (&mattr); +pthread_mutexattr_setpshared (&mattr,PTHREAD_PROCESS_SHARED ); +pthread_mutex_init (&sync->mutex, &mattr); +pthread_mutexattr_destroy (&mattr); + +pthread_condattr_init (&cattr); +pthread_condattr_setpshared (&cattr,PTHREAD_PROCESS_SHARED ); +pthread_cond_init (&sync->cond, &cattr); +pthread_condattr_destroy (&cattr); + + sync->step =STATE_INIT ; + +#define TEST_STATE_AWAIT (__state) \ + do { \ + pthread_mutex_lock(&sync->mutex); \ + while (sync->step != (__state)) { \ + struct timespec ts, stop; \ + int ret; \ + \ + clock_gettime(CLOCK_REALTIME, &ts); \ + stop = timespec_add_ns(ts, 100 * 1000000UL); \ + \ + ret = pthread_cond_timedwait(&sync->cond, &sync->mutex, &stop); \ + if (ret == ETIMEDOUT) { \ + bool alive = (child_pid == 0) ? \ + (getppid() == parent_pid) : \ + (waitpid(child_pid, NULL, WNOHANG) == 0); \ + TEST_ASSERT(alive, "Other process exited prematurely"); \ + } else { \ + TEST_ASSERT(!ret, "pthread_cond_timedwait failed"); \ + } \ + } \ + pthread_mutex_unlock(&sync->mutex); \ + } while (0) + +#define TEST_STATE_SET (__state) \ + do { \ + pthread_mutex_lock(&sync->mutex); \ + sync->step = (__state); \ + pthread_cond_broadcast(&sync->cond); \ + pthread_mutex_unlock(&sync->mutex); \ + } while (0) + + parent_pid =getpid (); + child_pid =fork (); +TEST_ASSERT (child_pid !=-1 ,"fork failed" ); + +if (child_pid ==0 ) { +const char inconsequential =0xdd ; + +TEST_STATE_AWAIT (STATE_CHECK_SHARED ); + +/* +* This maps the pages into the child process as well, and tests +* that the conversion process will unmap the guest_memfd memory +* from all processes. +*/ +host_do_rmw (t->mem,0 ,0xB ,0xC ); + +TEST_STATE_SET (STATE_DONE_CHECKING_SHARED ); +TEST_STATE_AWAIT (STATE_CHECK_PRIVATE ); + +TEST_EXPECT_SIGBUS (READ_ONCE (t->mem[0 ])); +TEST_EXPECT_SIGBUS (WRITE_ONCE (t->mem[0 ], inconsequential)); + +TEST_STATE_SET (STATE_DONE_CHECKING_PRIVATE ); +exit (0 ); + } + +test_shared (t,0 ,0 ,0xA ,0xB ); + +TEST_STATE_SET (STATE_CHECK_SHARED ); +TEST_STATE_AWAIT (STATE_DONE_CHECKING_SHARED ); + +test_convert_to_private (t,0 ,0xC ,0xD ); + +TEST_STATE_SET (STATE_CHECK_PRIVATE ); +TEST_STATE_AWAIT (STATE_DONE_CHECKING_PRIVATE ); + +TEST_ASSERT_EQ (waitpid (child_pid, &status,0 ), child_pid); +TEST_ASSERT (WIFEXITED (status) &&WEXITSTATUS (status) ==0 , +"Child exited with unexpected status" ); + +pthread_mutex_destroy (&sync->mutex); +pthread_cond_destroy (&sync->cond); +kvm_munmap (sync,sizeof (*sync)); + +#undef TEST_STATE_SET +#undef TEST_STATE_AWAIT +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 37/45 KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
In reply to Ackerley TngAdd helper functions to allow KVM selftests to pin memory using CONFIG_GUP_TEST. This is useful for creating test scenarios where some page has an increased refcount, such as when testing guest_memfd in-place conversion.
The helpers open /sys/kernel/debug/gup_test and invoke the PIN_LONGTERM_TEST_START and PIN_LONGTERM_TEST_STOP ioctls. Since this functionality depends on the kernel being built with CONFIG_GUP_TEST, provide stub implementations that trigger a test failure if the configuration is missing.
--- tools/testing/selftests/kvm/include/kvm_util.h | 3 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
@@ -1206,6 +1206,9 @@ static inline int pin_self_to_any_cpu(void)return pin_task_to_any_cpu (pthread_self ()); } +void pin_pages (void *vaddr,uint64_t size); +void unpin_pages (void ); +void kvm_print_vcpu_pinning_help (void );void kvm_parse_vcpu_pinning (const char *pcpus_string,u32 vcpu_to_pcpu[],int nr_vcpus);
tools/testing/selftests/kvm/lib/kvm_util.c +25 -0
@@ -19,6 +19,8 @@#include <unistd.h> #include <linux/kernel.h> +#include "../../../../../mm/gup_test.h" +#define KVM_UTIL_MIN_PFN 2u32 kvm_random_seed;
@@ -645,6 +647,29 @@ int __pin_task_to_cpu(pthread_t task, int cpu)return pthread_setaffinity_np (task,sizeof (cpuset), &cpuset); } +static int gup_test_fd =-1 ; + +void pin_pages (void *vaddr,uint64_t size) +{ +const struct pin_longterm_test args = { + .addr = (uint64_t )vaddr, + .size = size, + .flags =PIN_LONGTERM_TEST_FLAG_USE_WRITE , + }; + + gup_test_fd =__open_path_or_exit ("/sys/kernel/debug/gup_test" ,O_RDWR , +"Is CONFIG_GUP_TEST enabled?" ); + +TEST_ASSERT_EQ (ioctl (gup_test_fd,PIN_LONGTERM_TEST_START , &args),0 ); +} + +void unpin_pages (void ) +{ +TEST_ASSERT_EQ (ioctl (gup_test_fd,PIN_LONGTERM_TEST_STOP ),0 ); + +kvm_free_fd (gup_test_fd); +} +static u32 parse_pcpu (const char *cpu_str,const cpu_set_t *allowed_mask) {u32 pcpu =atoi_non_negative ("CPU number" , cpu_str);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 38/45 KVM: selftests: Test conversion with elevated page refcount
In reply to Ackerley TngAdd a selftest to verify that converting a shared guest_memfd page to a private page fails if the page has an elevated reference count.
When KVM converts a shared page to a private one, it expects the page to have a reference count equal to the reference counts taken by the filemap. If another kernel subsystem holds a reference to the page, the conversion must be aborted.
The test asserts that both bulk and single-page conversion attempts correctly fail with EAGAIN for the pinned page. After the page is unpinned, the test verifies that subsequent conversions succeed.
--- .../kvm/x86/guest_memfd_conversions_test.c | 56 ++++++++++++++++++++++ 1 file changed, 56 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +56 -0
@@ -444,6 +444,62 @@ GMEM_CONVERSION_TEST_INIT_SHARED(forked_accesses)#undef TEST_STATE_AWAIT } +static void test_convert_to_private_fails (test_data_t *t,u64 pgoff, +size_t nr_pages, +u64 expected_error_offset) +{ +/* +1 to make it anything but expected_error_offset. */ +u64 error_offset = expected_error_offset +1 ; +u64 offset = pgoff * page_size; +int ret; + +do { + ret =__gmem_set_private (t->gmem_fd, offset, + nr_pages * page_size, &error_offset); + }while (ret ==-1 && errno ==EINTR ); +TEST_ASSERT (ret ==-1 && errno ==EAGAIN , +"Wanted EAGAIN on page %lu, got %d (ret = %d)" , pgoff, + errno, ret); +TEST_ASSERT_EQ (error_offset, expected_error_offset); +} + +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(elevated_refcount, 4) +{ +int i; + +pin_pages (t->mem + test_page * page_size, page_size); + +for (i =0 ; i < nr_pages; i++) +test_shared (t, i,0 ,'A' ,'B' ); + +/* +* Converting in bulk should fail as long any page in the range has +* unexpected refcounts. +*/ +test_convert_to_private_fails (t,0 , nr_pages, test_page * page_size); + +for (i =0 ; i < nr_pages; i++) { +/* +* Converting page-wise should also fail as long any page in the +* range has unexpected refcounts. +*/ +if (i == test_page) +test_convert_to_private_fails (t, i,1 , test_page * page_size); +else +test_convert_to_private (t, i,'B' ,'C' ); + } + +unpin_pages (); + +gmem_set_private (t->gmem_fd,0 , nr_pages * page_size); + +for (i =0 ; i < nr_pages; i++) { +char expected = i == test_page ?'B' :'C' ; + +test_private (t, i, expected,'D' ); + } +} +int main (int argc,char *argv[]) {TEST_REQUIRE (kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (KVM_X86_SW_PROTECTED_VM ));
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 39/45 KVM: selftests: Reset shared memory after hole-punching
In reply to Ackerley Tngprivate_mem_conversions_test used to reset the shared memory that was used for the test to an initial pattern at the end of each test iteration. Then, it would punch out the pages, which would zero memory.
Without in-place conversion, the resetting would write shared memory, and hole-punching will zero private memory, hence resetting the test to the state at the beginning of the for loop.
With in-place conversion, resetting writes memory as shared, and hole-punching zeroes the same physical memory, hence undoing the reset done before the hole punch.
Move the resetting after the hole-punching, and reset the entire PER_CPU_DATA_SIZE instead of just the tested range.
With in-place conversion, this zeroes and then resets the same physical memory. Without in-place conversion, the private memory is zeroed, and the shared memory is reset to init_p.
This is sufficient since at each test stage, the memory is assumed to start as shared, and private memory is always assumed to start zeroed. Conversion zeroes memory, so the future test stages will work as expected.
--- tools/testing/selftests/kvm/x86/private_mem_conversions_test.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +6 -3
@@ -202,15 +202,18 @@ static void guest_test_explicit_conversion(u64 base_gpa, bool do_fallocate)guest_sync_shared (gpa, size, p3, p4);memcmp_g (gpa, p4, size); -/* Reset the shared memory back to the initial pattern. */ -memset ((void *)gpa, init_p, size); -/* * Free (via PUNCH_HOLE) *all* private memory so that the next * iteration starts from a clean slate, e.g. with respect to * whether or not there are pages/folios in guest_mem. */ guest_map_shared (base_gpa,PER_CPU_DATA_SIZE , true); + +/* +* Hole-punching above zeroed private memory. Reset shared +* memory in preparation for the next GUEST_STAGE. +*/ +memset ((void *)base_gpa, init_p,PER_CPU_DATA_SIZE ); } }
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 40/45 KVM: selftests: Provide function to look up guest_memfd details from gpa
In reply to Ackerley TngIntroduce a new helper, kvm_gpa_to_guest_memfd(), to find the guest_memfd-related details of a memory region that contains a given guest physical address (GPA).
The function returns the file descriptor for the memfd, the offset into the file that corresponds to the GPA, and the number of bytes remaining in the region from that GPA.
kvm_gpa_to_guest_memfd() was factored out from vm_guest_mem_fallocate(); refactor vm_guest_mem_fallocate() to use the new helper.
--- tools/testing/selftests/kvm/include/kvm_util.h | 3 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 37 ++++++++++++++++---------- 2 files changed, 26 insertions(+), 14 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
@@ -428,6 +428,9 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)vm_ioctl (vm,KVM_ENABLE_CAP , &enable_cap); } +int kvm_gpa_to_guest_memfd (struct kvm_vm *vm,gpa_t gpa,off_t *fd_offset, +size_t *nr_bytes); +/* * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes. These * flows need significant enhancements to support multiple attributes.
tools/testing/selftests/kvm/lib/kvm_util.c +23 -14
@@ -1332,27 +1332,20 @@ void vm_guest_mem_fallocate(struct kvm_vm *vm, u64 base, u64 size, bool punch_hole) {const int mode =FALLOC_FL_KEEP_SIZE | (punch_hole ?FALLOC_FL_PUNCH_HOLE :0 ); -struct userspace_mem_region *region;u64 end = base + size; -gpa_t gpa, len;off_t fd_offset; -int ret; +int fd, ret; +size_t len; +gpa_t gpa;for (gpa = base; gpa < end; gpa += len) { -u64 offset; - - region =userspace_mem_region_find (vm, gpa, gpa); -TEST_ASSERT (region && region->region.flags &KVM_MEM_GUEST_MEMFD , -"Private memory region not found for GPA 0x%lx" , gpa); + fd =kvm_gpa_to_guest_memfd (vm, gpa, &fd_offset, &len); + len =min (end - gpa, len); - offset = gpa - region->region.guest_phys_addr; - fd_offset = region->region.guest_memfd_offset + offset; - len =min_t (u64, end - gpa, region->region.memory_size - offset); - - ret =fallocate (region->region.guest_memfd, mode, fd_offset, len); + ret =fallocate (fd, mode, fd_offset, len);TEST_ASSERT (!ret,"fallocate() failed to %s at %lx (len = %lu), fd = %d, mode = %x, offset = %lx" , punch_hole ?"punch hole" :"allocate" , gpa, len, - region->region.guest_memfd, mode, fd_offset); + fd, mode, fd_offset); } }
@@ -1689,6 +1682,22 @@ void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa)return (void *) ((uintptr_t ) region->host_alias + offset); } +int kvm_gpa_to_guest_memfd (struct kvm_vm *vm,gpa_t gpa,off_t *fd_offset, +size_t *nr_bytes) +{ +struct userspace_mem_region *region; +gpa_t gpa_offset; + + region =userspace_mem_region_find (vm, gpa, gpa); +TEST_ASSERT (region && region->region.flags &KVM_MEM_GUEST_MEMFD , +"guest_memfd memory region not found for GPA 0x%lx" , gpa); + + gpa_offset = gpa - region->region.guest_phys_addr; + *fd_offset = region->region.guest_memfd_offset + gpa_offset; + *nr_bytes = region->region.memory_size - gpa_offset; +return region->region.guest_memfd; +} +/* Create an interrupt controller chip for the specified VM. */ void vm_create_irqchip (struct kvm_vm *vm) {
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 41/45 KVM: selftests: Provide common function to set memory attributes
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Introduce vm_mem_set_memory_attributes(), which handles setting of memory attributes for a range of guest physical addresses, regardless of whether the attributes should be set via guest_memfd or via the memory attributes at the VM level.
Refactor existing vm_mem_set_{shared,private} functions to use the new function. Opportunistically update the size parameter to use size_t instead of u64.
--- tools/testing/selftests/kvm/include/kvm_util.h | 46 +++++++++++++++++++------- 1 file changed, 34 insertions(+), 12 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +34 -12
@@ -454,18 +454,6 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,vm_ioctl (vm,KVM_SET_MEMORY_ATTRIBUTES , &attr); } -static inline void vm_mem_set_private (struct kvm_vm *vm,gpa_t gpa, -u64 size) -{ -vm_set_memory_attributes (vm, gpa, size,KVM_MEMORY_ATTRIBUTE_PRIVATE ); -} - -static inline void vm_mem_set_shared (struct kvm_vm *vm,gpa_t gpa, -u64 size) -{ -vm_set_memory_attributes (vm, gpa, size,0 ); -} -static inline int __gmem_set_memory_attributes (int fd,u64 offset,size_t size,u64 attributes,u64 *error_offset)
@@ -532,6 +520,40 @@ static inline void gmem_set_shared(int fd, u64 offset, size_t size)gmem_set_memory_attributes (fd, offset, size,0 ); } +static inline void vm_mem_set_memory_attributes (struct kvm_vm *vm,gpa_t gpa, +size_t size,u64 attrs) +{ +if (kvm_has_gmem_attributes) { +gpa_t end = gpa + size; +off_t fd_offset; +gpa_t addr; +size_t len; +int fd; + +for (addr = gpa; addr < end; addr += len) { + fd =kvm_gpa_to_guest_memfd (vm, addr, &fd_offset, &len); + len =min (end - addr, len); + +gmem_set_memory_attributes (fd, fd_offset, len, attrs); + } + }else { +vm_set_memory_attributes (vm, gpa, size, attrs); + } +} + +static inline void vm_mem_set_private (struct kvm_vm *vm,gpa_t gpa, +size_t size) +{ +vm_mem_set_memory_attributes (vm, gpa, size, +KVM_MEMORY_ATTRIBUTE_PRIVATE ); +} + +static inline void vm_mem_set_shared (struct kvm_vm *vm,gpa_t gpa, +size_t size) +{ +vm_mem_set_memory_attributes (vm, gpa, size,0 ); +} +void vm_guest_mem_fallocate (struct kvm_vm *vm,gpa_t gpa,u64 size,bool punch_hole);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 42/45 KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
In reply to Ackerley TngThe TEST_EXPECT_SIGBUS macro is not thread-safe as it uses a global sigjmp_buf and installs a global SIGBUS signal handler. If multiple threads execute the macro concurrently, they will race on installing the signal handler and stomp on other threads' jump buffers, leading to incorrect test behavior.
Make TEST_EXPECT_SIGBUS thread-safe with the following changes:
Share the KVM tests' global signal handler. sigaction() applies to all threads; without sharing a global signal handler, one thread may have removed the signal handler that another thread added, hence leading to unexpected signals.
The alternative of layering signal handlers was considered, but calling sigaction() within TEST_EXPECT_SIGBUS() necessarily creates a race. To avoid adding new setup and teardown routines to do sigaction() and keep usage of TEST_EXPECT_SIGBUS() simple, share the KVM tests' global signal handler.
Opportunistically rename report_unexpected_signal to catchall_signal_handler.
To continue to only expect SIGBUS within specific regions of code, use a thread-specific variable, expecting_sigbus, to replace installing and removing signal handlers.
Make the execution environment for the thread, sigjmp_buf, a thread-specific variable.
As part of TEST_EXPECT_SIGBUS(), assert the prerequisite for this setup, that the current signal handler is the catchall_signal_handler.
--- tools/testing/selftests/kvm/include/test_util.h | 32 +++++++++++++------------ tools/testing/selftests/kvm/lib/kvm_util.c | 18 ++++++++++---- tools/testing/selftests/kvm/lib/test_util.c | 7 ------ 3 files changed, 30 insertions(+), 27 deletions(-)
tools/testing/selftests/kvm/include/test_util.h +17 -15
@@ -83,21 +83,23 @@ do { \__builtin_unreachable (); \ }while (0 ) -extern sigjmp_buf expect_sigbus_jmpbuf; -void expect_sigbus_handler (int signum); - -#define TEST_EXPECT_SIGBUS (action) \ -do { \ - struct sigaction sa_old, sa_new = { \ - .sa_handler = expect_sigbus_handler, \ - }; \ - \ - sigaction(SIGBUS, &sa_new, &sa_old); \ - if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) { \ - action; \ - TEST_FAIL("'%s' should have triggered SIGBUS", #action); \ - } \ - sigaction(SIGBUS, &sa_old, NULL); \ +extern __thread sigjmp_buf expect_sigbus_jmpbuf; +extern __threadvolatile sig_atomic_t expecting_sigbus; +void catchall_signal_handler (int signum); + +#define TEST_EXPECT_SIGBUS (action) \ +do { \ + struct sigaction __sa = {}; \ + \ + TEST_ASSERT_EQ(sigaction(SIGBUS, NULL, &__sa), 0); \ + TEST_ASSERT_EQ(__sa.sa_handler, &catchall_signal_handler); \ + \ + expecting_sigbus = true; \ + if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) { \ + action; \ + TEST_FAIL("'%s' should have triggered SIGBUS", #action);\ + } \ + expecting_sigbus = false; \ } while (0)size_t parse_size (const char *size);
tools/testing/selftests/kvm/lib/kvm_util.c +13 -5
@@ -2316,13 +2316,20 @@ __weak void kvm_selftest_arch_init(void) { } -static void report_unexpected_signal (int signum) +__threadsigjmp_buf expect_sigbus_jmpbuf; +__threadvolatile sig_atomic_t expecting_sigbus; + +void catchall_signal_handler (int signum) { +switch (signum) { +case SIGBUS : { +if (expecting_sigbus) +siglongjmp (expect_sigbus_jmpbuf,1 ); + +TEST_FAIL ("Unexpected SIGBUS (%d)\n" , signum); + }#define KVM_CASE_SIGNUM (sig) \ case sig: TEST_FAIL("Unexpected " #sig " (%d)\n", signum) - -switch (signum) { -KVM_CASE_SIGNUM (SIGBUS );KVM_CASE_SIGNUM (SIGSEGV );KVM_CASE_SIGNUM (SIGILL );KVM_CASE_SIGNUM (SIGFPE );
@@ -2334,12 +2341,13 @@ static void report_unexpected_signal(int signum)void __attribute((constructor))kvm_selftest_init (void ) {struct sigaction sig_sa = { - .sa_handler = report_unexpected_signal, + .sa_handler = catchall_signal_handler, };/* Tell stdout not to buffer its content. */ setbuf (stdout,NULL ); + expecting_sigbus = false;sigaction (SIGBUS , &sig_sa,NULL );sigaction (SIGSEGV , &sig_sa,NULL );sigaction (SIGILL , &sig_sa,NULL );
tools/testing/selftests/kvm/lib/test_util.c +0 -7
@@ -19,13 +19,6 @@#include "test_util.h" #include "kvm_syscalls.h" -sigjmp_buf expect_sigbus_jmpbuf; - -void __attribute__((used))expect_sigbus_handler (int signum) -{ -siglongjmp (expect_sigbus_jmpbuf,1 ); -} -/* * Random number generator that is usable from guest code. This is the * Park-Miller LCG using standard constants.
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH v12 43/45 KVM: selftests: Set up page size and alignment independently for guest_memfd
In reply to Ackerley TngCurrently, vm_mem_add derives the backing source page size, alignment padding, and mmap size from the backing source type upfront before checking if guest_memfd is being mmapped.
With shared memory also mmap()-ed from guest_memfd, the alignment of the mmap-ed address needs to respect guest_memfd's backing page size.
Refactor the backing store setup to configure the backing source page size, alignment, mmap flags, and mmap offset directly for guest_memfd when it is mmapped, ignoring the backing source type.
Skip hugepage validation and anonymous memory madvise calls when mmapping from guest_memfd, since those are not applicable when mmapping guest_memfd.
--- tools/testing/selftests/kvm/lib/kvm_util.c | 74 ++++++++++++++++++------------ 1 file changed, 45 insertions(+), 29 deletions(-)
tools/testing/selftests/kvm/lib/kvm_util.c +45 -29
@@ -1034,11 +1034,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, {int ret;struct userspace_mem_region *region; -size_t backing_src_pagesz =get_backing_src_pagesz (src_type); -int mmap_flags =vm_mem_backing_src_alias (src_type)->flag;size_t mem_size = npages * vm->page_size; -off_t mmap_offset =0 ; -size_t alignment =1 ; +size_t backing_src_pagesz; +off_t mmap_offset; +bool is_gmem_mmap; +size_t alignment; +int mmap_flags;TEST_REQUIRE_SET_USER_MEMORY_REGION2 ();
@@ -1090,19 +1091,31 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,/* Allocate and initialize new mem region structure. */ region =calloc (1 ,sizeof (*region));TEST_ASSERT (region !=NULL ,"Insufficient Memory" ); - region->mmap_size = mem_size; -/* -* When using THP mmap is not guaranteed to returned a hugepage aligned -* address so we have to pad the mmap. Padding is not needed for HugeTLB -* because mmap will always return an address aligned to the HugeTLB -* page size. -*/ -if (src_type ==VM_MEM_SRC_ANONYMOUS_THP ) - alignment =max (backing_src_pagesz, alignment); + is_gmem_mmap = (flags &KVM_MEM_GUEST_MEMFD ) && + (gmem_flags &GUEST_MEMFD_FLAG_MMAP ); + +if (is_gmem_mmap) { + backing_src_pagesz =getpagesize (); + alignment =1 ; + mmap_flags =MAP_SHARED ; + mmap_offset = gmem_offset; + }else { + backing_src_pagesz =get_backing_src_pagesz (src_type); +/* +* When using THP mmap is not guaranteed to returned a hugepage aligned +* address so we have to pad the mmap. Padding is not needed for HugeTLB +* because mmap will always return an address aligned to the HugeTLB +* page size. +*/ + alignment = src_type ==VM_MEM_SRC_ANONYMOUS_THP ? backing_src_pagesz :1 ; + mmap_flags =vm_mem_backing_src_alias (src_type)->flag; + mmap_offset =0 ; + }TEST_ASSERT_EQ (gpa,align_up (gpa, backing_src_pagesz)); + region->mmap_size = mem_size;/* Add enough memory to align up if necessary */ if (alignment >1 ) region->mmap_size += alignment;
@@ -1129,10 +1142,8 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, } region->fd =-1 ; -if (flags &KVM_MEM_GUEST_MEMFD && gmem_flags &GUEST_MEMFD_FLAG_MMAP ) { +if (is_gmem_mmap) { region->fd =kvm_dup (gmem_fd); - mmap_flags =MAP_SHARED ; - mmap_offset = gmem_offset; }else if (backing_src_is_shared (src_type)) { region->fd =kvm_memfd_alloc (region->mmap_size, src_type ==VM_MEM_SRC_SHARED_HUGETLB );
@@ -1141,22 +1152,27 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->mmap_start =__kvm_mmap (region->mmap_size,PROT_READ |PROT_WRITE , mmap_flags, region->fd, mmap_offset); -TEST_ASSERT (!is_backing_src_hugetlb (src_type) || - region->mmap_start ==align_ptr_up (region->mmap_start, backing_src_pagesz), -"mmap_start %p is not aligned to HugeTLB page size 0x%lx" , - region->mmap_start, backing_src_pagesz); -/* Align host address */ region->host_mem =align_ptr_up (region->mmap_start, alignment); -/* As needed perform madvise */ -if ((src_type ==VM_MEM_SRC_ANONYMOUS || - src_type ==VM_MEM_SRC_ANONYMOUS_THP ) &&thp_configured ()) { - ret =madvise (region->host_mem, mem_size, - src_type ==VM_MEM_SRC_ANONYMOUS ?MADV_NOHUGEPAGE :MADV_HUGEPAGE ); -TEST_ASSERT (ret ==0 ,"madvise failed, addr: %p length: 0x%lx src_type: %s" , - region->host_mem, mem_size, -vm_mem_backing_src_alias (src_type)->name); +if (!is_gmem_mmap) { +TEST_ASSERT (!is_backing_src_hugetlb (src_type) || + region->mmap_start == +align_ptr_up (region->mmap_start, backing_src_pagesz), +"mmap_start %p is not aligned to HugeTLB page size 0x%lx" , + region->mmap_start, backing_src_pagesz); + +/* As needed perform madvise */ +if ((src_type ==VM_MEM_SRC_ANONYMOUS || + src_type ==VM_MEM_SRC_ANONYMOUS_THP ) &&thp_configured ()) { +int advice = src_type ==VM_MEM_SRC_ANONYMOUS ? +MADV_NOHUGEPAGE :MADV_HUGEPAGE ; + + ret =madvise (region->host_mem, mem_size, advice); +TEST_ASSERT (ret ==0 ,"madvise failed, addr: %p length: 0x%lx src_type: %s" , + region->host_mem, mem_size, +vm_mem_backing_src_alias (src_type)->name); + } } region->backing_src_type = src_type;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 43/45] KVM: selftests: Set up page size and alignment independently for guest_memfd
Quoted 10 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
Currently,
vm_mem_addderives the backing source page size, alignment padding, and mmap size from the backing source type upfront before checking ifguest_memfdis being mmapped.With shared memory also mmap()-ed from
guest_memfd, the alignment of the mmap-ed address needs to respect guest_memfd's backing page size.
Quoted 8 lines
Refactor the backing store setup to configure the backing source page size, alignment, mmap flags, and mmap offset directly for
guest_memfdwhen it is mmapped, ignoring the backing source type.Skip hugepage validation and anonymous memory madvise calls when mmapping from
guest_memfd, since those are not applicable when mmappingguest_memfd.Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 124 lines
--- tools/testing/selftests/kvm/lib/kvm_util.c | 74 ++++++++++++++++++------------ 1 file changed, 45 insertions(+), 29 deletions(-)tools/testing/selftests/kvm/lib/kvm_util.c +45
-30@@ -1034,11 +1034,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, {int ret;struct userspace_mem_region *region; -size_t backing_src_pagesz =get_backing_src_pagesz (src_type); -int mmap_flags =vm_mem_backing_src_alias (src_type)->flag;size_t mem_size = npages * vm->page_size; -off_t mmap_offset =0 ; -size_t alignment =1 ; +size_t backing_src_pagesz; +off_t mmap_offset; +bool is_gmem_mmap; +size_t alignment; +int mmap_flags;TEST_REQUIRE_SET_USER_MEMORY_REGION2 ();@@ -1090,19 +1091,31 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,/* Allocate and initialize new mem region structure. */ region =calloc (1 ,sizeof (*region));TEST_ASSERT (region !=NULL ,"Insufficient Memory" ); - region->mmap_size = mem_size; -/* -* When using THP mmap is not guaranteed to returned a hugepage aligned -* address so we have to pad the mmap. Padding is not needed for HugeTLB -* because mmap will always return an address aligned to the HugeTLB -* page size. -*/ -if (src_type ==VM_MEM_SRC_ANONYMOUS_THP ) - alignment =max (backing_src_pagesz, alignment); + is_gmem_mmap = (flags &KVM_MEM_GUEST_MEMFD ) && + (gmem_flags &GUEST_MEMFD_FLAG_MMAP ); + +if (is_gmem_mmap) { + backing_src_pagesz =getpagesize (); + alignment =1 ; + mmap_flags =MAP_SHARED ; + mmap_offset = gmem_offset; + }else { + backing_src_pagesz =get_backing_src_pagesz (src_type); +/* +* When using THP mmap is not guaranteed to returned a hugepage aligned +* address so we have to pad the mmap. Padding is not needed for HugeTLB +* because mmap will always return an address aligned to the HugeTLB +* page size. +*/ + alignment = src_type ==VM_MEM_SRC_ANONYMOUS_THP ? backing_src_pagesz :1 ; + mmap_flags =vm_mem_backing_src_alias (src_type)->flag; + mmap_offset =0 ; + }TEST_ASSERT_EQ (gpa,align_up (gpa, backing_src_pagesz)); + region->mmap_size = mem_size;/* Add enough memory to align up if necessary */ if (alignment >1 ) region->mmap_size += alignment;@@ -1129,10 +1142,8 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, } region->fd =-1 ; -if (flags &KVM_MEM_GUEST_MEMFD && gmem_flags &GUEST_MEMFD_FLAG_MMAP ) { +if (is_gmem_mmap) { region->fd =kvm_dup (gmem_fd); - mmap_flags =MAP_SHARED ; - mmap_offset = gmem_offset; }else if (backing_src_is_shared (src_type)) { region->fd =kvm_memfd_alloc (region->mmap_size, src_type ==VM_MEM_SRC_SHARED_HUGETLB );@@ -1141,22 +1152,27 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->mmap_start =__kvm_mmap (region->mmap_size,PROT_READ |PROT_WRITE , mmap_flags, region->fd, mmap_offset); -TEST_ASSERT (!is_backing_src_hugetlb (src_type) || - region->mmap_start ==align_ptr_up (region->mmap_start, backing_src_pagesz), -"mmap_start %p is not aligned to HugeTLB page size 0x%lx" , - region->mmap_start, backing_src_pagesz); -/* Align host address */ region->host_mem =align_ptr_up (region->mmap_start, alignment); -/* As needed perform madvise */ -if ((src_type ==VM_MEM_SRC_ANONYMOUS || - src_type ==VM_MEM_SRC_ANONYMOUS_THP ) &&thp_configured ()) { - ret =madvise (region->host_mem, mem_size, - src_type ==VM_MEM_SRC_ANONYMOUS ?MADV_NOHUGEPAGE :MADV_HUGEPAGE ); -TEST_ASSERT (ret ==0 ,"madvise failed, addr: %p length: 0x%lx src_type: %s" , - region->host_mem, mem_size, -vm_mem_backing_src_alias (src_type)->name); +if (!is_gmem_mmap) { +TEST_ASSERT (!is_backing_src_hugetlb (src_type) || + region->mmap_start == +align_ptr_up (region->mmap_start, backing_src_pagesz), +"mmap_start %p is not aligned to HugeTLB page size 0x%lx" , + region->mmap_start, backing_src_pagesz); + +/* As needed perform madvise */ +if ((src_type ==VM_MEM_SRC_ANONYMOUS || + src_type ==VM_MEM_SRC_ANONYMOUS_THP ) &&thp_configured ()) { +int advice = src_type ==VM_MEM_SRC_ANONYMOUS ? +MADV_NOHUGEPAGE :MADV_HUGEPAGE ; + + ret =madvise (region->host_mem, mem_size, advice); +TEST_ASSERT (ret ==0 ,"madvise failed, addr: %p length: 0x%lx src_type: %s" , + region->host_mem, mem_size, +vm_mem_backing_src_alias (src_type)->name); + } } region->backing_src_type = src_type; --2.55.0.897.gb25b4bd76c-goog
PATCH v12 44/45 KVM: selftests: Update private_mem_conversions_test for in-place conversions
In reply to Ackerley TngUpdate private_mem_conversions_test for in-place conversions. In-place conversions support is detected in selftests with kvm_has_gmem_attributes.
With in-place conversions, specifying userspace_addr from some other memory provider that is not the guest_memfd associated with the memslot is a user error, since KVM will only use both shared and private memory from the guest_memfd.
Hence, when kvm_has_gmem_attributes, only test in-place conversions with single backing, where guest_memfd provides both shared and private memory.
For single backing, guest_memfd must be created with GUEST_MEMFD_FLAG_MMAP. Initialize the guest_memfd as shared to align with how memory would default to shared when shared/private state was tracked at the VM level (the test expects this, it was written for non-in-place-conversions).
When handling a hypercall to set attributes, use vm_mem_set_memory_attributes() to send the ioctl to the guest_memfd instead of the VM.
When testing in-place conversions (single-backing), don't allow the user to configure src_type, since src_type will be ignored. Don't use src_type to determine alignment for rounding up per-cpu test memory size, since guest_memfd's backing page size is always the system PAGE_SIZE.
--- .../kvm/x86/private_mem_conversions_test.c | 57 +++++++++++++++++----- 1 file changed, 44 insertions(+), 13 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +44 -13
@@ -306,9 +306,12 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu)if (do_fallocate)vm_guest_mem_fallocate (vm, gpa, size, map_shared); -if (set_attributes) -vm_set_memory_attributes (vm, gpa, size, - map_shared ?0 :KVM_MEMORY_ATTRIBUTE_PRIVATE ); +if (set_attributes) { +u64 attrs = map_shared ?0 :KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +vm_mem_set_memory_attributes (vm, gpa, size, attrs); + } + run->hypercall.ret =0 ; }
@@ -352,8 +355,21 @@ static void *__test_mem_conversions(void *__vcpu)size_t nr_bytes =min_t (size_t, vm->page_size, size - i);u8 *hva =addr_gpa2hva (vm, gpa + i); -/* In all cases, the host should observe the shared data. */ -memcmp_h (hva, gpa + i, uc.args[3 ], nr_bytes); +if (kvm_has_gmem_attributes && + uc.args[0 ] ==SYNC_PRIVATE ) { +TEST_EXPECT_SIGBUS (READ_ONCE (*hva)); + }else { +/* +* If not testing in-place conversion +* (dual backing), the host should +* always observe shared data, since the +* shared memory is a separate page. +* +* For SYNC_SHARED, test that the host +* can see shared memory. +*/ +memcmp_h (hva, gpa + i, uc.args[3 ], nr_bytes); + }/* For shared, write the new pattern to guest memory. */ if (uc.args[0 ] ==SYNC_SHARED )
@@ -369,20 +385,29 @@ static void *__test_mem_conversions(void *__vcpu) } } +/* Align each vCPU's chunk of memory naturally to the size of the backing store. */ +static size_t compute_per_cpu_size (enum vm_mem_backing_src_type src_type) +{ +size_t alignment; + +if (kvm_has_gmem_attributes) + alignment =getpagesize (); +else + alignment =get_backing_src_pagesz (src_type); + +return align_up (PER_CPU_DATA_SIZE ,max_t (size_t,SZ_2M , alignment)); +} +static void test_mem_conversions (enum vm_mem_backing_src_type src_type,u32 nr_vcpus,u32 nr_memslots) { -/* -* Allocate enough memory so that each vCPU's chunk of memory can be -* naturally aligned with respect to the size of the backing store. -*/ -const size_t alignment =max_t (size_t,SZ_2M ,get_backing_src_pagesz (src_type)); -const size_t per_cpu_size =align_up (PER_CPU_DATA_SIZE , alignment); +const size_t per_cpu_size =compute_per_cpu_size (src_type);const size_t memfd_size = per_cpu_size * nr_vcpus;const size_t slot_size = memfd_size / nr_memslots;struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ];pthread_t threads[KVM_MAX_VCPUS ];struct kvm_vm *vm; +u64 gmem_flags;int memfd, i;const struct vm_shape shape = {
@@ -397,12 +422,16 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_vvm_enable_cap (vm,KVM_CAP_EXIT_HYPERCALL , (1 <<KVM_HC_MAP_GPA_RANGE )); - memfd =vm_create_guest_memfd (vm, memfd_size,0 ); + gmem_flags =0 ; +if (kvm_has_gmem_attributes) + gmem_flags =GUEST_MEMFD_FLAG_MMAP |GUEST_MEMFD_FLAG_INIT_SHARED ; + + memfd =vm_create_guest_memfd (vm, memfd_size, gmem_flags);for (i =0 ; i < nr_memslots; i++)vm_mem_add (vm, src_type,BASE_DATA_GPA + slot_size * i,BASE_DATA_SLOT + i, slot_size / vm->page_size, -KVM_MEM_GUEST_MEMFD , memfd, slot_size * i,0 ); +KVM_MEM_GUEST_MEMFD , memfd, slot_size * i, gmem_flags);for (i =0 ; i < nr_vcpus; i++) {gpa_t gpa =BASE_DATA_GPA + i * per_cpu_size;
@@ -462,6 +491,8 @@ int main(int argc, char *argv[])while ((opt =getopt (argc, argv,"hm:s:n:" )) !=-1 ) {switch (opt) {case 's' : +TEST_ASSERT (!kvm_has_gmem_attributes, +"src_type is only configurable when testing without in-place conversion" ); src_type =parse_backing_src_type (optarg);break ;case 'n' :
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Re: [PATCH v12 44/45] KVM: selftests: Update private_mem_conversions_test for in-place conversions
Quoted 24 lines · On Mon, 31 Aug 2026 at 01:25, Ackerley Tng via B4 Relay…
From: Ackerley Tng <[email protected]>
Update
private_mem_conversions_testfor in-place conversions. In-place conversions support is detected in selftests withkvm_has_gmem_attributes.With in-place conversions, specifying
userspace_addrfrom some other memory provider that is not theguest_memfdassociated with the memslot is a user error, since KVM will only use both shared and private memory from theguest_memfd.Hence, when
kvm_has_gmem_attributes, only test in-place conversions with single backing, whereguest_memfdprovides both shared and private memory.For single backing,
guest_memfdmust be created withGUEST_MEMFD_FLAG_MMAP. Initialize theguest_memfdas shared to align with how memory would default to shared when shared/private state was tracked at the VM level (the test expects this, it was written for non-in-place-conversions).When handling a hypercall to set attributes, use
vm_mem_set_memory_attributes()to send the ioctl to theguest_memfdinstead of the VM.
Quoted 8 lines
When testing in-place conversions (single-backing), don't allow the user to configure
src_type, sincesrc_typewill be ignored. Don't usesrc_typeto determine alignment for rounding up per-cpu test memory size, since guest_memfd's backing page size is always the systemPAGE_SIZE.Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ackerley Tng <[email protected]>
Cheers,
/fuad
Quoted 117 lines
--- .../kvm/x86/private_mem_conversions_test.c | 57 +++++++++++++++++----- 1 file changed, 44 insertions(+), 13 deletions(-)tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +44
-14@@ -306,9 +306,12 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu)if (do_fallocate)vm_guest_mem_fallocate (vm, gpa, size, map_shared); -if (set_attributes) -vm_set_memory_attributes (vm, gpa, size, - map_shared ?0 :KVM_MEMORY_ATTRIBUTE_PRIVATE ); +if (set_attributes) { +u64 attrs = map_shared ?0 :KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +vm_mem_set_memory_attributes (vm, gpa, size, attrs); + } + run->hypercall.ret =0 ; }@@ -352,8 +355,21 @@ static void *__test_mem_conversions(void *__vcpu)size_t nr_bytes =min_t (size_t, vm->page_size, size - i);u8 *hva =addr_gpa2hva (vm, gpa + i); -/* In all cases, the host should observe the shared data. */ -memcmp_h (hva, gpa + i, uc.args[3 ], nr_bytes); +if (kvm_has_gmem_attributes && + uc.args[0 ] ==SYNC_PRIVATE ) { +TEST_EXPECT_SIGBUS (READ_ONCE (*hva)); + }else { +/* +* If not testing in-place conversion +* (dual backing), the host should +* always observe shared data, since the +* shared memory is a separate page. +* +* For SYNC_SHARED, test that the host +* can see shared memory. +*/ +memcmp_h (hva, gpa + i, uc.args[3 ], nr_bytes); + }/* For shared, write the new pattern to guest memory. */ if (uc.args[0 ] ==SYNC_SHARED )@@ -369,20 +385,29 @@ static void *__test_mem_conversions(void *__vcpu) } } +/* Align each vCPU's chunk of memory naturally to the size of the backing store. */ +static size_t compute_per_cpu_size (enum vm_mem_backing_src_type src_type) +{ +size_t alignment; + +if (kvm_has_gmem_attributes) + alignment =getpagesize (); +else + alignment =get_backing_src_pagesz (src_type); + +return align_up (PER_CPU_DATA_SIZE ,max_t (size_t,SZ_2M , alignment)); +} +static void test_mem_conversions (enum vm_mem_backing_src_type src_type,u32 nr_vcpus,u32 nr_memslots) { -/* -* Allocate enough memory so that each vCPU's chunk of memory can be -* naturally aligned with respect to the size of the backing store. -*/ -const size_t alignment =max_t (size_t,SZ_2M ,get_backing_src_pagesz (src_type)); -const size_t per_cpu_size =align_up (PER_CPU_DATA_SIZE , alignment); +const size_t per_cpu_size =compute_per_cpu_size (src_type);const size_t memfd_size = per_cpu_size * nr_vcpus;const size_t slot_size = memfd_size / nr_memslots;struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ];pthread_t threads[KVM_MAX_VCPUS ];struct kvm_vm *vm; +u64 gmem_flags;int memfd, i;const struct vm_shape shape = {@@ -397,12 +422,16 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_vvm_enable_cap (vm,KVM_CAP_EXIT_HYPERCALL , (1 <<KVM_HC_MAP_GPA_RANGE )); - memfd =vm_create_guest_memfd (vm, memfd_size,0 ); + gmem_flags =0 ; +if (kvm_has_gmem_attributes) + gmem_flags =GUEST_MEMFD_FLAG_MMAP |GUEST_MEMFD_FLAG_INIT_SHARED ; + + memfd =vm_create_guest_memfd (vm, memfd_size, gmem_flags);for (i =0 ; i < nr_memslots; i++)vm_mem_add (vm, src_type,BASE_DATA_GPA + slot_size * i,BASE_DATA_SLOT + i, slot_size / vm->page_size, -KVM_MEM_GUEST_MEMFD , memfd, slot_size * i,0 ); +KVM_MEM_GUEST_MEMFD , memfd, slot_size * i, gmem_flags);for (i =0 ; i < nr_vcpus; i++) {gpa_t gpa =BASE_DATA_GPA + i * per_cpu_size;@@ -462,6 +491,8 @@ int main(int argc, char *argv[])while ((opt =getopt (argc, argv,"hm:s:n:" )) !=-1 ) {switch (opt) {case 's' : +TEST_ASSERT (!kvm_has_gmem_attributes, +"src_type is only configurable when testing without in-place conversion" ); src_type =parse_backing_src_type (optarg);break ;case 'n' : --2.55.0.897.gb25b4bd76c-goog
PATCH v12 45/45 KVM: selftests: Update private memory exits test to work with per-gmem attributes
In reply to Ackerley TngFrom: Sean Christopherson <[email protected]>
Skip setting memory to private in the private memory exits test when using per-gmem memory attributes, as memory is initialized to private by default for guest_memfd, and using vm_mem_set_private() on a guest_memfd instance requires creating guest_memfd with GUEST_MEMFD_FLAG_MMAP (which is totally doable, but would need to be conditional and is ultimately unnecessary).
Expect an emulated MMIO instead of a memory fault exit when attributes are per-gmem, as deleting the memslot effectively drops the private status, i.e. the GPA becomes shared and thus supports emulated MMIO.
Skip the "memslot not private" test entirely, as private vs. shared state for x86 software-protected VMs comes from the memory attributes themselves, and so when doing in-place conversions there can never be a disconnect between the expected and actual states.
--- .../selftests/kvm/x86/private_mem_kvm_exits_test.c | 36 ++++++++++++++++++---- 1 file changed, 30 insertions(+), 6 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c +30 -6
@@ -62,8 +62,9 @@ static void test_private_access_memslot_deleted(void)virt_map (vm,EXITS_TEST_GVA ,EXITS_TEST_GPA ,EXITS_TEST_NPAGES ); -/* Request to access page privately */ -vm_mem_set_private (vm,EXITS_TEST_GPA ,EXITS_TEST_SIZE ); +/* Request to access page privately. */ +if (!kvm_has_gmem_attributes) +vm_mem_set_private (vm,EXITS_TEST_GPA ,EXITS_TEST_SIZE );kvm_pthread_create (&vm_thread,NULL , (pthread_fn_t )run_vcpu_get_exit_reason, (void *)vcpu);
@@ -73,10 +74,26 @@ static void test_private_access_memslot_deleted(void)kvm_pthread_join (vm_thread, &thread_return); exit_reason = (u32 )(u64 )thread_return; -TEST_ASSERT_EQ (exit_reason,KVM_EXIT_MEMORY_FAULT ); -TEST_ASSERT_EQ (vcpu->run->memory_fault.flags,KVM_MEMORY_EXIT_FLAG_PRIVATE ); -TEST_ASSERT_EQ (vcpu->run->memory_fault.gpa,EXITS_TEST_GPA ); -TEST_ASSERT_EQ (vcpu->run->memory_fault.size,EXITS_TEST_SIZE ); +/* +* If attributes are tracked per-gmem, deleting the memslot that points +* at the gmem instance effectively makes the memory shared, and so the +* read should trigger emulated MMIO. +* +* If attributes are tracked per-VM, deleting the memslot shouldn't +* affect the private attribute, and so KVM should generate a memory +* fault exit (emulated MMIO on private GPAs is disallowed). +*/ +if (kvm_has_gmem_attributes) { +TEST_ASSERT_EQ (exit_reason,KVM_EXIT_MMIO ); +TEST_ASSERT_EQ (vcpu->run->mmio.phys_addr,EXITS_TEST_GPA ); +TEST_ASSERT_EQ (vcpu->run->mmio.len,sizeof (u64)); +TEST_ASSERT_EQ (vcpu->run->mmio.is_write, false); + }else { +TEST_ASSERT_EQ (exit_reason,KVM_EXIT_MEMORY_FAULT ); +TEST_ASSERT_EQ (vcpu->run->memory_fault.flags,KVM_MEMORY_EXIT_FLAG_PRIVATE ); +TEST_ASSERT_EQ (vcpu->run->memory_fault.gpa,EXITS_TEST_GPA ); +TEST_ASSERT_EQ (vcpu->run->memory_fault.size,EXITS_TEST_SIZE ); + }kvm_vm_free (vm); }
@@ -87,6 +104,13 @@ static void test_private_access_memslot_not_private(void)struct kvm_vcpu *vcpu;u32 exit_reason; +/* +* Accessing non-private memory as private with a software-protected VM +* isn't possible when doing in-place conversions. +*/ +if (kvm_has_gmem_attributes) +return ; + vm =vm_create_shape_with_one_vcpu (protected_vm_shape, &vcpu, guest_repeatedly_read);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog