Skip to contentSkip to conversation

guest_memfd: In-place conversion support

PATCH Superseded Ackerley Tng 11 participants 145 messages Aug 7 – Aug 27 Page 1 of 2

Subsystems (13)

v10 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v10 cover guest_memfd: In-place conversion support

This is v10 of guest_memfd in-place conversion support, which addresses comments on v9. Please find the motivation for this series from v8 and before.

v9 is based on kvm-x86/next, with a cherry-picked version of David's GUP fix [1], and a cherry-picked version of David's refactoring patch [2].

Here's everything stitched together for your convenience:

https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v10

I tested v10 with CoCo selftests on top of the branch above:

https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-coco-selftests-v10

Tested with both CONFIG_KVM_VM_MEMORY_ATTRIBUTES enabled and disabled:

+ tools/testing/selftests/kvm/guest_memfd_test.c
+ tools/testing/selftests/kvm/pre_fault_memory_test.c
+ tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_conversions_test.c
+ tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c
+ The TDX and SNP conversions selftests in https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-coco-selftests-v10

[1] https://lore.kernel.org/all/20260731-check_and_migrate_movable_folios-v1-1-e0002d7b791e@kernel.org/
("mm/gup: fix always draining LRU caches in collect_longterm_unpinnable_folios()")
[2] https://lore.kernel.org/all/[email protected]/
("mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()")

Older series:

v9: https://lore.kernel.org/r/[email protected]
v8: https://lore.kernel.org/r/[email protected]
v7: https://lore.kernel.org/r/[email protected]
v6: https://lore.kernel.org/r/[email protected]
RFC v5: https://lore.kernel.org/r/[email protected]
RFC v4: https://lore.kernel.org/all/[email protected]/T/
RFC v3: https://lore.kernel.org/r/[email protected]/T/
RFC v2: https://lore.kernel.org/all/[email protected]/T/
RFC v1: https://lore.kernel.org/all/[email protected]/T/

Previous versions of this feature, part of other series, are available at:

+ https://lore.kernel.org/all/bd163de3118b626d1005aa88e71ef2fb72f0be0f.1726009989.git.ackerleytng@google.com/
+ https://lore.kernel.org/all/[email protected]/
+ https://lore.kernel.org/all/b784326e9ccae6a08388f1bf39db70a2204bdc51.1747264138.git.ackerleytng@google.com/

---
Ackerley Tng (21):
KVM: guest_memfd: Filter both shared and private when invalidating
KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2
KVM: guest_memfd: Ensure pages are not in use before conversion
KVM: guest_memfd: Call arch make_shared callback for to-shared conversion
KVM: guest_memfd: Return early if range already has requested attributes
KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check
KVM: guest_memfd: Zero page while getting pfn
KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION
KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes
KVM: selftests: Test basic single-page conversion flow
KVM: selftests: Test conversion flow when INIT_SHARED
KVM: selftests: Test conversion precision in guest_memfd
KVM: selftests: Test conversion before allocation
KVM: selftests: Convert with allocated folios in different layouts
KVM: selftests: Test that truncation does not change shared/private status
KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
KVM: selftests: Test conversion with elevated page refcount
KVM: selftests: Reset shared memory after hole-punching
KVM: selftests: Provide function to look up guest_memfd details from gpa
KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

David Hildenbrand (Arm) (1):
mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()

Michael Roth (1):
KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE

Sean Christopherson (18):
KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory
KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings
KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES
KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined
KVM: Rename memory attribute APIs to prepare for in-place gmem conversion
KVM: Provide generic interface for checking memory private/shared status
KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion
KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h
KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86
KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs
KVM: selftests: Create gmem fd before "regular" fd when adding memslot
KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}
KVM: selftests: Add support for mmap() on guest_memfd in core library
KVM: selftests: Add selftests global for guest memory attributes capability
KVM: selftests: Add helpers for calling ioctls on guest_memfd
KVM: selftests: Test that shared/private status is consistent across processes
KVM: selftests: Provide common function to set memory attributes
KVM: selftests: Update private memory exits test to work with per-gmem attributes

 Documentation/virt/kvm/api.rst                     |  78 +++-
 .../virt/kvm/x86/amd-memory-encryption.rst         |  14 +-
 Documentation/virt/kvm/x86/intel-tdx.rst           |   4 +
 arch/x86/include/asm/kvm-x86-ops.h                 |   2 +-
 arch/x86/include/asm/kvm_host.h                    |   6 +-
 arch/x86/kvm/Kconfig                               |  15 +-
 arch/x86/kvm/mmu/mmu.c                             |  16 +-
 arch/x86/kvm/svm/sev.c                             |  11 +-
 arch/x86/kvm/vmx/tdx.c                             |   8 +-
 arch/x86/kvm/x86.c                                 |  20 +-
 include/linux/kvm_host.h                           |  76 +--
 include/linux/swap.h                               |  11 +-
 include/trace/events/kvm.h                         |   6 +-
 include/uapi/linux/kvm.h                           |  16 +
 mm/gup.c                                           |  15 +-
 mm/swap.c                                          |  48 ++
 tools/testing/selftests/kvm/Makefile.kvm           |   1 +
 tools/testing/selftests/kvm/include/kvm_util.h     | 139 +++++-
 tools/testing/selftests/kvm/include/test_util.h    |  34 +-
 tools/testing/selftests/kvm/lib/kvm_util.c         | 164 ++++---
 tools/testing/selftests/kvm/lib/test_util.c        |   7 -
 .../kvm/x86/guest_memfd_conversions_test.c         | 512 +++++++++++++++++++++
 .../kvm/x86/private_mem_conversions_test.c         |  53 ++-
 .../selftests/kvm/x86/private_mem_kvm_exits_test.c |  36 +-
 virt/kvm/Kconfig                                   |   3 -
 virt/kvm/guest_memfd.c                             | 450 ++++++++++++++++--
 virt/kvm/kvm_main.c                                |  92 ++--
 27 files changed, 1574 insertions(+), 263 deletions(-)

---
base-commit: 4217d4987bbe33e331764a752721f781123b2e07
change-id: 20260225-gmem-inplace-conversion-bd0dbd39753a

Best regards,
--
Ackerley Tng <[email protected]>

#1 lore raw

PATCH v10 1/41 KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory

From: Sean Christopherson <[email protected]>

Using kvm_mem_is_private() when populating guest_memfd instead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.

No functional change intended.

---
 virt/kvm/guest_memfd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/guest_memfd.c +1 -3
index b596486d184ca..a11cf6dc785da 100644
@@ -811,9 +811,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 
 	folio_unlock(folio);
 
-	if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+	if (!kvm_mem_is_private(kvm, gfn)) {
 		ret = -EINVAL;
 		goto out_put_folio;
 	}
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 1/41
#2 lore raw

Re: [PATCH v10 01/41] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory

Quoted 3 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Using kvm_mem_is_private() when populating guest_memfd instead of using an

Nit:
Using -> Use?

Quoted 9 lines

open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.

No functional change intended.

Quoted 20 lines
---
 virt/kvm/guest_memfd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/guest_memfd.c +1 -3
index b596486d184ca..a11cf6dc785da 100644
@@ -811,9 +811,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 
 	folio_unlock(folio);
 
-	if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+	if (!kvm_mem_is_private(kvm, gfn)) {
 		ret = -EINVAL;
 		goto out_put_folio;
 	}
Reviewed-by on v10 1/41
#3 lore raw

Re: [PATCH v10 01/41] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory

In reply to Ackerley Tng
Quoted 7 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Using kvm_mem_is_private() when populating guest_memfd instead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.

Quoted 5 lines

No functional change intended.

Quoted 20 lines
---
  virt/kvm/guest_memfd.c | 4 +---
  1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/guest_memfd.c +1 -3
index b596486d184ca..a11cf6dc785da 100644
@@ -811,9 +811,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 
  	folio_unlock(folio);
 
-	if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE,
-					     KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+	if (!kvm_mem_is_private(kvm, gfn)) {
  		ret = -EINVAL;
  		goto out_put_folio;
  	}
Reviewed-by on v10 1/41
#4 lore raw

Re: [PATCH v10 01/41] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory

In reply to Ackerley Tng
Quoted 8 lines · On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay…

From: Sean Christopherson <[email protected]>

Using kvm_mem_is_private() when populating guest_memfd instead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well.

Quoted 5 lines

No functional change intended.

Cheers,
/fuad

Quoted 24 lines
---
 virt/kvm/guest_memfd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/guest_memfd.c +1 -4
index b596486d184ca..a11cf6dc785da 100644
@@ -811,9 +811,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 
        folio_unlock(folio);
 
-       if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
-                                            KVM_MEMORY_ATTRIBUTE_PRIVATE,
-                                            KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+       if (!kvm_mem_is_private(kvm, gfn)) {
                ret = -EINVAL;
                goto out_put_folio;
        }
 
--

2.55.0.654.g21b8a5bc05-goog

#5 lore raw

PATCH v10 2/41 KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Start plumbing in guest_memfd support for in-place private<=>shared conversions by tracking attributes via a maple tree. KVM currently tracks private vs. shared attributes on a per-VM basis, which made sense when a guest_memfd _only_ supported private memory, but tracking per-VM simply can't work for in-place conversions as the shared/private status of a given page needs to be per-gmem_inode, not per-VM.

Use the filemap invalidation lock to protect the maple tree, as taking the lock for read when faulting in memory (for userspace or the guest) isn't expected to result in meaningful contention, and using a separate lock would add significant complexity (avoiding deadlock is quite difficult).

---
 virt/kvm/guest_memfd.c | 133 ++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 116 insertions(+), 17 deletions(-)
virt/kvm/guest_memfd.c +116 -17
index a11cf6dc785da..1f1ae61b4878b 100644
@@ -4,6 +4,7 @@
 #include <linux/falloc.h>
 #include <linux/fs.h>
 #include <linux/kvm_host.h>
+#include <linux/maple_tree.h>
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
@@ -34,6 +35,13 @@ struct gmem_inode {
 	struct list_head gmem_file_list;
 
 	u64 flags;
+	/*
+	 * Every index in this inode, whether memory is populated or
+	 * not, is tracked in attributes. The entire range of indices,
+	 * corresponding to the size of this inode, is represented in
+	 * this maple tree.
+	 */
+	struct maple_tree attributes;
 };
 
 static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
@@ -61,9 +69,25 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
 	return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
+static u64 kvm_gmem_get_attributes(struct inode *inode, void *entry)
+{
+	if (WARN_ON_ONCE(!entry)) {
+		bool initially_shared = GMEM_I(inode)->flags &
+					GUEST_MEMFD_FLAG_INIT_SHARED;
+
+		return initially_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
+	}
+
+	return xa_to_value(entry);
+}
+
 static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index)
 {
-	return !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED);
+	struct maple_tree *mt = &GMEM_I(inode)->attributes;
+	void *entry = mtree_load(mt, index);
+
+	return kvm_gmem_get_attributes(inode, entry) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 
 static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)
@@ -364,10 +388,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
 	if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode))
 		return VM_FAULT_SIGBUS;
 
-	if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff))
-		return VM_FAULT_SIGBUS;
+	filemap_invalidate_lock_shared(inode->i_mapping);
+	if (kvm_gmem_is_shared_mem(inode, vmf->pgoff))
+		folio = kvm_gmem_get_folio(inode, vmf->pgoff);
+	else
+		folio = ERR_PTR(-EACCES);
+	filemap_invalidate_unlock_shared(inode->i_mapping);
 
-	folio = kvm_gmem_get_folio(inode, vmf->pgoff);
 	if (IS_ERR(folio)) {
 		if (PTR_ERR(folio) == -EAGAIN)
 			return VM_FAULT_RETRY;
@@ -520,6 +547,51 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
+static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	MA_STATE(mas, &gi->attributes, 0, (size >> PAGE_SHIFT) - 1);
+	u64 attrs;
+	int r;
+
+	inode->i_op = &kvm_gmem_iops;
+	inode->i_mapping->a_ops = &kvm_gmem_aops;
+	inode->i_mode |= S_IFREG;
+	inode->i_size = size;
+	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
+
+	/*
+	 * guest_memfd memory is neither migratable nor swappable: set
+	 * inaccessible to gate off both.
+	 */
+	mapping_set_inaccessible(inode->i_mapping);
+	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
+
+	gi->flags = flags;
+
+	mt_set_external_lock(&gi->attributes,
+			     &inode->i_mapping->invalidate_lock);
+
+	/*
+	 * Store default attributes for the entire gmem instance. Ensuring every
+	 * index is represented in the maple tree at all times simplifies the
+	 * conversion and merging logic.
+	 */
+	attrs = gi->flags & GUEST_MEMFD_FLAG_INIT_SHARED ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	/*
+	 * Acquire the invalidation lock purely to make lockdep happy.  The
+	 * maple tree library expects all stores to be protected via the lock,
+	 * and the library can't know when the tree is reachable only by the
+	 * caller, as is the case here.
+	 */
+	filemap_invalidate_lock(inode->i_mapping);
+	r = mas_store_gfp(&mas, xa_mk_value(attrs), GFP_KERNEL);
+	filemap_invalidate_unlock(inode->i_mapping);
+
+	return r;
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
@@ -550,16 +622,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 		goto err_fops;
 	}
 
-	inode->i_op = &kvm_gmem_iops;
-	inode->i_mapping->a_ops = &kvm_gmem_aops;
-	inode->i_mode |= S_IFREG;
-	inode->i_size = size;
-	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
-	mapping_set_inaccessible(inode->i_mapping);
-	/* Unmovable mappings are supposed to be marked unevictable as well. */
-	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
-
-	GMEM_I(inode)->flags = flags;
+	err = kvm_gmem_init_inode(inode, size, flags);
+	if (err)
+		goto err_inode;
 
 	file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops);
 	if (IS_ERR(file)) {
@@ -764,9 +829,13 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (!file)
 		return -EFAULT;
 
+	filemap_invalidate_lock_shared(file_inode(file)->i_mapping);
+
 	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
-	if (IS_ERR(folio))
-		return PTR_ERR(folio);
+	if (IS_ERR(folio)) {
+		r = PTR_ERR(folio);
+		goto out;
+	}
 
 	if (!folio_test_uptodate(folio)) {
 		clear_highpage(folio_page(folio, 0));
@@ -786,6 +855,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	else
 		folio_put(folio);
 
+out:
+	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
 	return r;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn);
@@ -915,6 +986,15 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 
 	mpol_shared_policy_init(&gi->policy, NULL);
 
+	/*
+	 * Memory attributes are protected by the filemap invalidation lock, but
+	 * the lock structure isn't available at this time.  Immediately mark
+	 * maple tree as using external locking so that accessing the tree
+	 * before it's fully initialized results in NULL pointer dereferences
+	 * and not more subtle bugs.
+	 */
+	mt_init_flags(&gi->attributes, MT_FLAGS_LOCK_EXTERN | MT_FLAGS_USE_RCU);
+
 	gi->flags = 0;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
@@ -922,7 +1002,26 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 
 static void kvm_gmem_destroy_inode(struct inode *inode)
 {
-	mpol_free_shared_policy(&GMEM_I(inode)->policy);
+	struct gmem_inode *gi = GMEM_I(inode);
+
+	mpol_free_shared_policy(&gi->policy);
+
+	/*
+	 * Note!  Checking for an empty tree is functionally necessary
+	 * to avoid explosions if the tree hasn't been fully
+	 * initialized, i.e. if the inode is being destroyed before
+	 * guest_memfd can set the external lock, lockdep would find
+	 * that the tree's internal ma_lock was not held.
+	 */
+	if (!mtree_empty(&gi->attributes)) {
+		/*
+		 * Acquire the invalidation lock purely to make lockdep happy,
+		 * the inode is unreachable at this point.
+		 */
+		filemap_invalidate_lock(inode->i_mapping);
+		__mt_destroy(&gi->attributes);
+		filemap_invalidate_unlock(inode->i_mapping);
+	}
 }
 
 static void kvm_gmem_free_inode(struct inode *inode)
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#6 lore raw

Re: [PATCH v10 02/41] KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings

Review static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 78

Quoted 12 lines · Ackerley Tng

static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)

(quoted diff) +7 -0
@@ -61,9 +69,25 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
 	return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
+static u64 kvm_gmem_get_attributes(struct inode *inode, void *entry)
+{
+	if (WARN_ON_ONCE(!entry)) {
+		bool initially_shared = GMEM_I(inode)->flags &
+					GUEST_MEMFD_FLAG_INIT_SHARED;
+
+		return initially_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;

As David pointed out in v9[*], a helper would make this code in particular much easier to read. Though my vote is to add "get" and make it plural, because this function is also plural. I.e. this as fixup:

virt/kvm/guest_memfd.c +10 -7
index f2bec0d49747..b7435a8d2405 100644
@@ -69,14 +69,17 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
        return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
+static u64 kvm_gmem_get_default_attributes(struct inode *inode)
+{
+       bool init_shared = GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED;
+
+       return init_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+
 static u64 kvm_gmem_get_attributes(struct inode *inode, void *entry)
 {
-       if (WARN_ON_ONCE(!entry)) {
-               bool initially_shared = GMEM_I(inode)->flags &
-                                       GUEST_MEMFD_FLAG_INIT_SHARED;
-
-               return initially_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
-       }
+       if (WARN_ON_ONCE(!entry))
+               return kvm_gmem_get_default_attributes(inode);
 
        return xa_to_value(entry);
 }
@@ -833,7 +836,7 @@ static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags)
         * index is represented in the maple tree at all times simplifies the
         * conversion and merging logic.
         */
-       attrs = gi->flags & GUEST_MEMFD_FLAG_INIT_SHARED ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
+       attrs = kvm_gmem_get_default_attributes(inode);
 
        /*
         * Acquire the invalidation lock purely to make lockdep happy.  The

[*] https://lore.kernel.org/all/[email protected]

Reviewed-by on v10 2/41
#7 lore raw

Re: [PATCH v10 02/41] KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings

In reply to Ackerley Tng
Quoted 14 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Start plumbing in guest_memfd support for in-place private<=>shared conversions by tracking attributes via a maple tree. KVM currently tracks private vs. shared attributes on a per-VM basis, which made sense when a guest_memfd _only_ supported private memory, but tracking per-VM simply can't work for in-place conversions as the shared/private status of a given page needs to be per-gmem_inode, not per-VM.

Use the filemap invalidation lock to protect the maple tree, as taking the lock for read when faulting in memory (for userspace or the guest) isn't expected to result in meaningful contention, and using a separate lock would add significant complexity (avoiding deadlock is quite difficult).

Quoted 9 lines

With Sean's fixup in https://lore.kernel.org/kvm/[email protected]/

#8 lore raw

PATCH v10 3/41 KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Rename the per-VM memory attributes Kconfig to make it explicitly about per-VM attributes in anticipation of adding memory attributes support to guest_memfd, at which point it will be possible (and desirable) to have memory attributes without the per-VM support, even in x86.

No functional change intended.

---
 arch/x86/include/asm/kvm_host.h |  2 +-
 arch/x86/kvm/Kconfig            |  6 +++---
 arch/x86/kvm/mmu/mmu.c          |  2 +-
 arch/x86/kvm/x86.c              |  2 +-
 include/linux/kvm_host.h        |  8 ++++----
 include/trace/events/kvm.h      |  4 ++--
 virt/kvm/Kconfig                |  2 +-
 virt/kvm/kvm_main.c             | 14 +++++++-------
 8 files changed, 20 insertions(+), 20 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index 283847619ff88..14d86851fce78 100644
@@ -1851,7 +1851,7 @@ enum kvm_intr_type {
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest && \
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
 
arch/x86/kvm/Kconfig +3 -3
index 538ed1e803326..2f8e1b5a8a72c 100644
@@ -84,7 +84,7 @@ config KVM_SW_PROTECTED_VM
 	bool "Enable support for KVM software-protected VMs"
 	depends on EXPERT
 	depends on KVM_X86 && X86_64
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
+	select KVM_VM_MEMORY_ATTRIBUTES
 	help
 	  Enable support for KVM software-protected VMs.  Currently, software-
 	  protected VMs are purely a development and testing vehicle for
@@ -135,7 +135,7 @@ config KVM_INTEL_TDX
 	bool "Intel Trust Domain Extensions (TDX) support"
 	default y
 	depends on INTEL_TDX_HOST
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
+	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
 	  Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -159,7 +159,7 @@ config KVM_AMD_SEV
 	depends on KVM_AMD && X86_64
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
+	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/mmu/mmu.c +1 -1
index c519e8e8d646f..acafa7964964c 100644
@@ -8070,7 +8070,7 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 		vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
 				int level)
 {
arch/x86/kvm/x86.c +1 -1
index d94b59140c452..f1d977060489b 100644
@@ -10134,7 +10134,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		}
 	}
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	kvm_mmu_init_memslot_memory_attributes(kvm, slot);
 #endif
 
include/linux/kvm_host.h +4 -4
index 03bfc92864b6e..3e86cdd00e59b 100644
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 }
 #endif
 
-#ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 {
 	return false;
@@ -872,7 +872,7 @@ struct kvm {
 #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
 	struct notifier_block pm_notifier;
 #endif
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	/* Protected by slots_lock (for writes) and RCU (for reads) */
 	struct xarray mem_attr_array;
 #endif
@@ -2560,7 +2560,7 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot)
 	return slot->flags & KVM_MEMSLOT_GMEM_ONLY;
 }
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
 	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
@@ -2582,7 +2582,7 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
 }
-#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
+#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
include/trace/events/kvm.h +2 -2
index b282e3a867696..1ba72bd73ea2f 100644
@@ -358,7 +358,7 @@ TRACE_EVENT(kvm_dirty_ring_exit,
 	TP_printk("vcpu %d", __entry->vcpu_id)
 );
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 /*
  * @start:	Starting address of guest memory range
  * @end:	End address of guest memory range
@@ -383,7 +383,7 @@ TRACE_EVENT(kvm_vm_set_mem_attributes,
 	TP_printk("%#016llx -- %#016llx [0x%lx]",
 		  __entry->start, __entry->end, __entry->attr)
 );
-#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
+#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 TRACE_EVENT(kvm_unmap_hva_range,
 	TP_PROTO(unsigned long start, unsigned long end),
virt/kvm/Kconfig +1 -1
index c3c0ee253fc77..df575449d97b5 100644
@@ -100,7 +100,7 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG
 config KVM_MMU_LOCKLESS_AGING
        bool
 
-config KVM_GENERIC_MEMORY_ATTRIBUTES
+config KVM_VM_MEMORY_ATTRIBUTES
        bool
 
 config KVM_GUEST_MEMFD
virt/kvm/kvm_main.c +7 -7
index f3a6ba740ff79..e3a77d1e6d8ae 100644
@@ -1116,7 +1116,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 	spin_lock_init(&kvm->mn_invalidate_lock);
 	rcuwait_init(&kvm->mn_memslots_update_rcuwait);
 	xa_init(&kvm->vcpu_array);
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	xa_init(&kvm->mem_attr_array);
 #endif
 
@@ -1301,7 +1301,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 	cleanup_srcu_struct(&kvm->irq_srcu);
 	srcu_barrier(&kvm->srcu);
 	cleanup_srcu_struct(&kvm->srcu);
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	xa_destroy(&kvm->mem_attr_array);
 #endif
 	kvm_arch_free_vm(kvm);
@@ -2419,7 +2419,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
 	if (!kvm || kvm_arch_has_private_mem(kvm))
@@ -2624,7 +2624,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
 	return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes);
 }
-#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
+#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn)
 {
@@ -4941,7 +4941,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 	case KVM_CAP_SYSTEM_EVENT_DATA:
 	case KVM_CAP_DEVICE_CTRL:
 		return 1;
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	case KVM_CAP_MEMORY_ATTRIBUTES:
 		return kvm_supported_mem_attributes(kvm);
 #endif
@@ -5345,7 +5345,7 @@ static long kvm_vm_ioctl(struct file *filp,
 		break;
 	}
 #endif /* CONFIG_HAVE_KVM_IRQ_ROUTING */
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	case KVM_SET_MEMORY_ATTRIBUTES: {
 		struct kvm_memory_attributes attrs;
 
@@ -5356,7 +5356,7 @@ static long kvm_vm_ioctl(struct file *filp,
 		r = kvm_vm_ioctl_set_mem_attributes(kvm, &attrs);
 		break;
 	}
-#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
+#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 	case KVM_CREATE_DEVICE: {
 		struct kvm_create_device cd;
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#9 lore raw

PATCH v10 4/41 KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based on kvm_arch_has_private_mem being #defined in anticipation of tracking PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place conversion).

guest_memfd support for memory attributes is expected to be unconditional to avoid yet more macros (all architectures that support guest_memfd are expected to use per-gmem attributes at some point), at which point enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory attributes being supported by KVM at-large would result in a system-scope check (NULL @kvm) over-reporting support on arm64.

Give architectures full control over overriding the default definition of kvm_arch_has_private_mem() by removing the coupling with CONFIG_KVM_VM_MEMORY_ATTRIBUTES.

In a later patch, kvm_arch_has_private_mem() will be defined based on whether architectural features are compiled in, and made orthogonal to CONFIG_KVM_VM_MEMORY_ATTRIBUTES.

---
 include/linux/kvm_host.h | 2 +-
 virt/kvm/kvm_main.c      | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +1 -1
index 3e86cdd00e59b..e3666d0b40b74 100644
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 }
 #endif
 
-#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+#ifndef kvm_arch_has_private_mem
 static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 {
 	return false;
virt/kvm/kvm_main.c +2 -0
index e3a77d1e6d8ae..560ff05e95959 100644
@@ -2422,8 +2422,10 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
+#ifdef kvm_arch_has_private_mem
 	if (!kvm || kvm_arch_has_private_mem(kvm))
 		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
+#endif
 
 	return 0;
 }
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 4/41
#10 lore raw

Re: [PATCH v10 04/41] KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined

Quoted 22 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based on kvm_arch_has_private_mem being #defined in anticipation of tracking PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place conversion).

guest_memfd support for memory attributes is expected to be unconditional to avoid yet more macros (all architectures that support guest_memfd are expected to use per-gmem attributes at some point), at which point enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory attributes being supported by KVM at-large would result in a system-scope check (NULL @kvm) over-reporting support on arm64.

Give architectures full control over overriding the default definition of kvm_arch_has_private_mem() by removing the coupling with CONFIG_KVM_VM_MEMORY_ATTRIBUTES.

In a later patch, kvm_arch_has_private_mem() will be defined based on whether architectural features are compiled in, and made orthogonal to CONFIG_KVM_VM_MEMORY_ATTRIBUTES.

Quoted 5 lines
Quoted 34 lines
---
  include/linux/kvm_host.h | 2 +-
  virt/kvm/kvm_main.c      | 2 ++
  2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +1 -1
index 3e86cdd00e59b..e3666d0b40b74 100644
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
  }
  #endif
 
-#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+#ifndef kvm_arch_has_private_mem
  static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
  	return false;
virt/kvm/kvm_main.c +2 -0
index e3a77d1e6d8ae..560ff05e95959 100644
@@ -2422,8 +2422,10 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
  #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
  static u64 kvm_supported_mem_attributes(struct kvm *kvm)
  {
+#ifdef kvm_arch_has_private_mem
  	if (!kvm || kvm_arch_has_private_mem(kvm))
  		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
+#endif
 
  	return 0;
  }
#11 lore raw

PATCH v10 5/41 KVM: Rename memory attribute APIs to prepare for in-place gmem conversion

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Rename memory attribute APIs to add a "vm_" in the name in anticipation of moving PRIVATE tracking into guest_memfd, to allow in-place conversion between SHARED and PRIVATE. At that point, there will effectively be two (potential) sources of memory attributes: the VM and guest_memfd.

kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it is a helper local to the file.) Update the accompanying trace function to match.

No functional change intended.

---
 arch/x86/kvm/mmu/mmu.c     | 14 +++++++-------
 include/linux/kvm_host.h   | 16 ++++++++--------
 include/trace/events/kvm.h |  2 +-
 virt/kvm/kvm_main.c        | 32 ++++++++++++++++----------------
 4 files changed, 32 insertions(+), 32 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -7
index acafa7964964c..2b3bc73845a8a 100644
@@ -8089,8 +8089,8 @@ static void hugepage_set_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
 	lpage_info_slot(gfn, slot, level)->disallow_lpage |= KVM_LPAGE_MIXED_FLAG;
 }
 
-bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
-					struct kvm_gfn_range *range)
+bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
+					   struct kvm_gfn_range *range)
 {
 	struct kvm_memory_slot *slot = range->slot;
 	int level;
@@ -8159,18 +8159,18 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,
 	const unsigned long end = start + KVM_PAGES_PER_HPAGE(level);
 
 	if (level == PG_LEVEL_2M)
-		return kvm_range_has_memory_attributes(kvm, start, end, ~0, attrs);
+		return kvm_range_has_vm_memory_attributes(kvm, start, end, ~0, attrs);
 
 	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1)) {
 		if (hugepage_test_mixed(slot, gfn, level - 1) ||
-		    attrs != kvm_get_memory_attributes(kvm, gfn))
+		    attrs != kvm_get_vm_memory_attributes(kvm, gfn))
 			return false;
 	}
 	return true;
 }
 
-bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
-					 struct kvm_gfn_range *range)
+bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
+					    struct kvm_gfn_range *range)
 {
 	unsigned long attrs = range->arg.attributes;
 	struct kvm_memory_slot *slot = range->slot;
@@ -8263,7 +8263,7 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 		 * be manually checked as the attributes may already be mixed.
 		 */
 		for (gfn = start; gfn < end; gfn += nr_pages) {
-			unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+			unsigned long attrs = kvm_get_vm_memory_attributes(kvm, gfn);
 
 			if (hugepage_has_attrs(kvm, slot, gfn, level, attrs))
 				hugepage_clear_mixed(slot, gfn, level);
include/linux/kvm_host.h +8 -8
index e3666d0b40b74..3464a0ddbae74 100644
@@ -2561,21 +2561,21 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot)
 }
 
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
+static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
 	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
 }
 
-bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
-				     unsigned long mask, unsigned long attrs);
-bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
-					struct kvm_gfn_range *range);
-bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
-					 struct kvm_gfn_range *range);
+bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					unsigned long mask, unsigned long attrs);
+bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
+					   struct kvm_gfn_range *range);
+bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
+					    struct kvm_gfn_range *range);
 
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
-	return kvm_get_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+	return kvm_get_vm_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 #else
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
include/trace/events/kvm.h +1 -1
index 1ba72bd73ea2f..c4d514d2350cf 100644
@@ -364,7 +364,7 @@ TRACE_EVENT(kvm_dirty_ring_exit,
  * @end:	End address of guest memory range
  * @attr:	The value of the attribute being set.
  */
-TRACE_EVENT(kvm_vm_set_mem_attributes,
+TRACE_EVENT(kvm_set_vm_mem_attributes,
 	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr),
 	TP_ARGS(start, end, attr),
 
virt/kvm/kvm_main.c +16 -16
index 560ff05e95959..b3f9396123be2 100644
@@ -2420,7 +2420,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
 
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-static u64 kvm_supported_mem_attributes(struct kvm *kvm)
+static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 #ifdef kvm_arch_has_private_mem
 	if (!kvm || kvm_arch_has_private_mem(kvm))
@@ -2434,19 +2434,19 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
  * Returns true if _all_ gfns in the range [@start, @end) have attributes
  * such that the bits in @mask match @attrs.
  */
-bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
-				     unsigned long mask, unsigned long attrs)
+bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					unsigned long mask, unsigned long attrs)
 {
 	XA_STATE(xas, &kvm->mem_attr_array, start);
 	unsigned long index;
 	void *entry;
 
-	mask &= kvm_supported_mem_attributes(kvm);
+	mask &= kvm_supported_vm_mem_attributes(kvm);
 	if (attrs & ~mask)
 		return false;
 
 	if (end == start + 1)
-		return (kvm_get_memory_attributes(kvm, start) & mask) == attrs;
+		return (kvm_get_vm_memory_attributes(kvm, start) & mask) == attrs;
 
 	guard(rcu)();
 	if (!attrs)
@@ -2517,8 +2517,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 		KVM_MMU_UNLOCK(kvm);
 }
 
-static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
-					  struct kvm_gfn_range *range)
+static bool kvm_pre_set_vm_memory_attributes(struct kvm *kvm,
+					     struct kvm_gfn_range *range)
 {
 	/*
 	 * Unconditionally add the range to the invalidation set, regardless of
@@ -2533,18 +2533,18 @@ static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
 	 */
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
 
-	return kvm_arch_pre_set_memory_attributes(kvm, range);
+	return kvm_arch_pre_set_vm_memory_attributes(kvm, range);
 }
 
 /* Set @attributes for the gfn range [@start, @end). */
-static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long attributes)
 {
 	struct kvm_mmu_notifier_range pre_set_range = {
 		.start = start,
 		.end = end,
 		.arg.attributes = attributes,
-		.handler = kvm_pre_set_memory_attributes,
+		.handler = kvm_pre_set_vm_memory_attributes,
 		.on_lock = kvm_mmu_invalidate_start,
 		.flush_on_ret = true,
 		.may_block = true,
@@ -2553,7 +2553,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		.start = start,
 		.end = end,
 		.arg.attributes = attributes,
-		.handler = kvm_arch_post_set_memory_attributes,
+		.handler = kvm_arch_post_set_vm_memory_attributes,
 		.on_lock = kvm_mmu_invalidate_end,
 		.may_block = true,
 	};
@@ -2563,12 +2563,12 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	entry = attributes ? xa_mk_value(attributes) : NULL;
 
-	trace_kvm_vm_set_mem_attributes(start, end, attributes);
+	trace_kvm_set_vm_mem_attributes(start, end, attributes);
 
 	mutex_lock(&kvm->slots_lock);
 
 	/* Nothing to do if the entire range has the desired attributes. */
-	if (kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes))
+	if (kvm_range_has_vm_memory_attributes(kvm, start, end, ~0, attributes))
 		goto out_unlock;
 
 	/*
@@ -2607,7 +2607,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	/* flags is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
-	if (attrs->attributes & ~kvm_supported_mem_attributes(kvm))
+	if (attrs->attributes & ~kvm_supported_vm_mem_attributes(kvm))
 		return -EINVAL;
 	if (attrs->size == 0 || attrs->address + attrs->size < attrs->address)
 		return -EINVAL;
@@ -2624,7 +2624,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	 */
 	BUILD_BUG_ON(sizeof(attrs->attributes) != sizeof(unsigned long));
 
-	return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes);
+	return kvm_set_vm_mem_attributes(kvm, start, end, attrs->attributes);
 }
 #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
@@ -4945,7 +4945,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 		return 1;
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	case KVM_CAP_MEMORY_ATTRIBUTES:
-		return kvm_supported_mem_attributes(kvm);
+		return kvm_supported_vm_mem_attributes(kvm);
 #endif
 #ifdef CONFIG_KVM_GUEST_MEMFD
 	case KVM_CAP_GUEST_MEMFD:
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 5/41
#12 lore raw

Re: [PATCH v10 05/41] KVM: Rename memory attribute APIs to prepare for in-place gmem conversion

Quoted 13 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Rename memory attribute APIs to add a "vm_" in the name in anticipation of moving PRIVATE tracking into guest_memfd, to allow in-place conversion between SHARED and PRIVATE. At that point, there will effectively be two (potential) sources of memory attributes: the VM and guest_memfd.

kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it is a helper local to the file.) Update the accompanying trace function to match.

Quoted 7 lines

No functional change intended.

#13 lore raw

PATCH v10 6/41 KVM: Provide generic interface for checking memory private/shared status

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Introduce a generic kvm_mem_is_private() interface using a static call to determine if a GFN is private. This allows the implementation for checking a GFN's private/shared status to be set at runtime.

In preparation for choosing implementations between a guest_memfd lookup and the existing VM attribute lookup, rename the existing VM-attribute-based check to kvm_vm_mem_is_private to emphasize that it looks up VM attributes.

---
 include/linux/kvm_host.h | 14 ++++++++++++--
 virt/kvm/kvm_main.c      | 15 +++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)
include/linux/kvm_host.h +12 -2
index 3464a0ddbae74..65fbce46b63f4 100644
@@ -2573,16 +2573,26 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 					    struct kvm_gfn_range *range);
 
-static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return kvm_get_vm_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
+#endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
+
+#ifdef kvm_arch_has_private_mem
+typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
+DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
+
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return static_call(__kvm_mem_is_private)(kvm, gfn);
+}
 #else
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
 }
-#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
+#endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
virt/kvm/kvm_main.c +15 -0
index b3f9396123be2..a01d336eab18b 100644
@@ -2628,6 +2628,20 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
+#ifdef kvm_arch_has_private_mem
+DEFINE_STATIC_CALL_RET0(__kvm_mem_is_private, kvm_mem_is_private_t);
+EXPORT_STATIC_CALL_GPL(__kvm_mem_is_private);
+
+static void kvm_init_memory_attributes(void)
+{
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+	static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
+#endif
+}
+#else
+static void kvm_init_memory_attributes(void) { }
+#endif
+
 struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn)
 {
 	return __gfn_to_memslot(kvm_memslots(kvm), gfn);
@@ -6547,6 +6561,7 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
 	kvm_preempt_ops.sched_in = kvm_sched_in;
 	kvm_preempt_ops.sched_out = kvm_sched_out;
 
+	kvm_init_memory_attributes();
 	kvm_init_debug();
 
 	r = kvm_vfio_ops_init();
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 6/41
#14 lore raw

Re: [PATCH v10 06/41] KVM: Provide generic interface for checking memory private/shared status

Quoted 9 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Introduce a generic kvm_mem_is_private() interface using a static call to determine if a GFN is private. This allows the implementation for checking a GFN's private/shared status to be set at runtime.

In preparation for choosing implementations between a guest_memfd lookup and the existing VM attribute lookup, rename the existing VM-attribute-based check to kvm_vm_mem_is_private to emphasize that it

Nit:
kvm_vm_mem_is_private -> kvm_vm_mem_is_private()

Quoted 7 lines

looks up VM attributes.

#15 lore raw

PATCH v10 7/41 KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

---
 include/linux/kvm_host.h |  6 ++++++
 virt/kvm/guest_memfd.c   | 26 ++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      | 12 +++++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +6 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;
+
 typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
 DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 	return static_call(__kvm_mem_is_private)(kvm, gfn);
 }
 #else
+#define gmem_in_place_conversion false
+
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);
+
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
 		     int *max_order);
virt/kvm/guest_memfd.c +26 -0
index 1f1ae61b4878b..75979c885e030 100644
@@ -475,6 +475,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 	return 0;
 }
 
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+	struct inode *inode;
+
+	if (!slot)
+		return 0;
+
+	CLASS(gmem_get_file, file)(slot);
+	if (!file)
+		return 0;
+
+	inode = file_inode(file);
+
+	/*
+	 * Rely on the maple tree's internal RCU lock to ensure a stable result.
+	 * This result can become stale as soon as the lock is dropped, so the
+	 * caller _must_ protect consumption of private vs. shared either by
+	 * holding guest_memfd's invalidate lock for the entire duration, or by
+	 * checking mmu_invalidate_retry_gfn() under mmu_lock to serialize
+	 * against concurrent attribute updates.
+	 */
+	return kvm_gmem_is_private_mem(inode, kvm_gmem_get_index(slot, gfn));
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private);
+
 static struct file_operations kvm_gmem_fops = {
 	.mmap		= kvm_gmem_mmap,
 	.open		= generic_file_open,
virt/kvm/kvm_main.c +11 -1
index a01d336eab18b..0a5a1bee8de0f 100644
@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);
 static bool __ro_after_init allow_unsafe_mappings;
 module_param(allow_unsafe_mappings, bool, 0444);
 
+#ifdef kvm_arch_has_private_mem
+bool __ro_after_init gmem_in_place_conversion = false;
+#endif
+
 /*
  * Ordering of locks:
  *
@@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 #ifdef kvm_arch_has_private_mem
+	if (gmem_in_place_conversion)
+		return 0;
+
 	if (!kvm || kvm_arch_has_private_mem(kvm))
 		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 #endif
@@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_mem_is_private);
 
 static void kvm_init_memory_attributes(void)
 {
+	if (gmem_in_place_conversion)
+		static_call_update(__kvm_mem_is_private, kvm_gmem_is_private);
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
+	else
+		static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
 #endif
 }
 #else
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#16 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 24 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

---
 include/linux/kvm_host.h |  6 ++++++
 virt/kvm/guest_memfd.c   | 26 ++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      | 12 +++++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +0 -0
index 65fbce46b63f4..9477ecebbbced 100644

include/linux/kvm_host.h static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2583

Quoted 4 lines
include/linux/kvm_host.h +1 -0
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;

Is there a "supports/has/enable" in there? And should we call it "kvm_gmem" for completeness?

Quoted 14 lines
include/linux/kvm_host.h +3 -0
+
 typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
 DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 	return static_call(__kvm_mem_is_private)(kvm, gfn);
 }
 #else
+#define gmem_in_place_conversion false
+
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2602

Quoted 4 lines
include/linux/kvm_host.h +1 -0
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);

As mentioned in v9, the name is confusing.

kvm_gmem_mem_is_private

?

Signature · 3 lines
Cheers,

David
#17 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:
include/linux/kvm_host.h +0 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
Quoted 5 lines

include/linux/kvm_host.h static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2583

Quoted 2 lines
include/linux/kvm_host.h +1 -0
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;

Is there a "supports/has/enable" in there? And should we call it "kvm_gmem" for completeness?

It's kinda stupid and definitely more than a bit inconsistent, but overall I think I actually like "gmem_in_place_conversion" the best.

gmem_has_in_place_conversion and gmem_supports_in_place_conversion are misleading because it's not just that guest_memfd has/supports in-place conversion, it's that that KVM is tracking PRIVATE in guest_memfd and so in-place conversion is the only option.

On the other hand, while gmem_in_place_conversion_enabled is better, it's not quite accurate either because userspace isn't strictly required to do in-place conversion.

As for a kvm_ prefix, IMO gmem_ is sufficient for a namespace, and not having kvm_ is consistent with most module params in KVM.

So I 100% agree this is an odd name and has all kinds of inconsistenties, but all in all I actually like the vagueness and the brevity.

Quoted 15 lines
Quoted 15 lines
include/linux/kvm_host.h +2 -0
 typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
 DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 	return static_call(__kvm_mem_is_private)(kvm, gfn);
 }
 #else
+#define gmem_in_place_conversion false
+
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
Quoted 8 lines

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2602

Quoted 2 lines
include/linux/kvm_host.h +1 -0
 #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);

As mentioned in v9, the name is confusing.

kvm_gmem_mem_is_private

?

Ya, +1 to kvm_gmem_mem_is_private.

#18 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 6 lines · Sean Christopherson
Quoted 5 lines

As mentioned in v9, the name is confusing.

kvm_gmem_mem_is_private

?

Ya, +1 to kvm_gmem_mem_is_private.

So kvm_gmem_mem_is_private() calls kvm_gmem_is_private_mem() internallly. Ummmm.

#19 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 10 lines · Xiaoyao Li
Quoted 6 lines · Sean Christopherson
Quoted 5 lines

As mentioned in v9, the name is confusing.

kvm_gmem_mem_is_private

?

Ya, +1 to kvm_gmem_mem_is_private.

So kvm_gmem_mem_is_private() calls kvm_gmem_is_private_mem() internallly. Ummmm.

How about

kvm_gmem_mem_is_private(kvm, gfn)
__gmem_mem_is_private(inode, index) <== this is a static function
anyway, though it calls kvm_gmem_get_attributes(inode, entry)

or

kvm_gmem_gfn_is_private(kvm, gfn)
kvm_gmem_is_private_mem(inode, index)

(rename the vm stuff with the suffix gfn_is_private())

or (current, it is actually weird)

kvm_gmem_mem_is_private(kvm, gfn)
kvm_gmem_is_private_mem(inode, index)

or

kvm_gmem_mem_is_private(kvm, gfn)
kvm_gmem_index_is_private(inode, index)

#20 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 13 lines · Ackerley Tng
Quoted 10 lines · Xiaoyao Li
Quoted 6 lines · Sean Christopherson
Quoted 5 lines

As mentioned in v9, the name is confusing.

kvm_gmem_mem_is_private

?

Ya, +1 to kvm_gmem_mem_is_private.

So kvm_gmem_mem_is_private() calls kvm_gmem_is_private_mem() internallly. Ummmm.

Quoted 6 lines

How about

kvm_gmem_mem_is_private(kvm, gfn)
__gmem_mem_is_private(inode, index) <== this is a static function
anyway, though it calls kvm_gmem_get_attributes(inode, entry)

Hmm, what if we do:

kvm_gmem_is_private_gfn(kvm, gfn)
kvm_gmem_is_private_mem(inode, index)

And then to be consistent with the non-gmem code, slot in the rename you suggest below:

KVM: Rename memory attribute APIs to prepare for in-place gmem conversion <rename kvm_mem_is_private() => kvm_is_private_gfn()>
KVM: Provide generic interface for checking memory private/shared status

I would also be a-ok with:

	kvm_gmem_is_private_mem(kvm, gfn)
	__kvm_gmem_is_private_mem(inode, index);

along with a kvm_mem_is_private() => kvm_is_private_mem() rename, though the more I look at this, the more I prefer scoping the external API to a gfn. Especially for the guest_memfd code, where KVM needs to query shared vs. private without a gfn.

I don't love kvm_gmem_mem_is_private(), it looks like a typo (though as above, I do like it more than kvm_gmem_is_private()).

Quoted 16 lines

or

kvm_gmem_gfn_is_private(kvm, gfn)
kvm_gmem_is_private_mem(inode, index)

(rename the vm stuff with the suffix gfn_is_private())

or (current, it is actually weird)

kvm_gmem_mem_is_private(kvm, gfn)
kvm_gmem_is_private_mem(inode, index)

or

kvm_gmem_mem_is_private(kvm, gfn)
kvm_gmem_index_is_private(inode, index)

#21 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

In reply to Sean Christopherson
Quoted 23 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:
include/linux/kvm_host.h +0 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
Quoted 5 lines

include/linux/kvm_host.h static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2583

Quoted 2 lines
include/linux/kvm_host.h +1 -0
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;

Is there a "supports/has/enable" in there? And should we call it "kvm_gmem" for completeness?

It's kinda stupid and definitely more than a bit inconsistent, but overall I think I actually like "gmem_in_place_conversion" the best.

gmem_has_in_place_conversion and gmem_supports_in_place_conversion are misleading because it's not just that guest_memfd has/supports in-place conversion, it's that that KVM is tracking PRIVATE in guest_memfd and so in-place conversion is the only option.

Quoted 6 lines

On the other hand, while gmem_in_place_conversion_enabled is better, it's not quite accurate either because userspace isn't strictly required to do in-place conversion.

As for a kvm_ prefix, IMO gmem_ is sufficient for a namespace, and not having kvm_ is consistent with most module params in KVM.

Maybe we should have a helper function instead of accessing the module parameter directly, then the gmem_in_place_conversion could just stay file-local and kvm_gmem_in_place_conversion() would be used by other code that wants to obtain the value.

Instead of the

#define mem_in_place_conversion false

We'd have

#define kvm_gmem_in_place_conversion() false

just a thought ...

Signature · 3 lines
Cheers,

David
#22 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:
include/linux/kvm_host.h +0 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
Quoted 5 lines

include/linux/kvm_host.h static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2583

Quoted 2 lines
include/linux/kvm_host.h +1 -0
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;

Is there a "supports/has/enable" in there? And should we call it "kvm_gmem" for completeness?

It's kinda stupid and definitely more than a bit inconsistent, but overall I think I actually like "gmem_in_place_conversion" the best.

gmem_has_in_place_conversion and gmem_supports_in_place_conversion are misleading because it's not just that guest_memfd has/supports in-place conversion, it's that that KVM is tracking PRIVATE in guest_memfd and so in-place conversion is the only option.

Quoted 6 lines

On the other hand, while gmem_in_place_conversion_enabled is better, it's not quite accurate either because userspace isn't strictly required to do in-place conversion.

As for a kvm_ prefix, IMO gmem_ is sufficient for a namespace, and not having kvm_ is consistent with most module params in KVM.

Maybe we should have a helper function instead of accessing the module parameter directly, then the gmem_in_place_conversion could just stay file-local and kvm_gmem_in_place_conversion() would be used by other code that wants to obtain the value.

Instead of the

Quoted 7 lines

#define mem_in_place_conversion false

We'd have

#define kvm_gmem_in_place_conversion() false

just a thought ...

FWIW, I'd rather prefix kvm_ than add a wrapper to get a boolean.

#23 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 41 lines · Sean Christopherson
Quoted 40 lines · David Hildenbrand (Arm)
Quoted 23 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:
include/linux/kvm_host.h +0 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
Quoted 5 lines

include/linux/kvm_host.h static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2583

Quoted 2 lines
include/linux/kvm_host.h +1 -0
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;

Is there a "supports/has/enable" in there? And should we call it "kvm_gmem" for completeness?

It's kinda stupid and definitely more than a bit inconsistent, but overall I think I actually like "gmem_in_place_conversion" the best.

gmem_has_in_place_conversion and gmem_supports_in_place_conversion are misleading because it's not just that guest_memfd has/supports in-place conversion, it's that that KVM is tracking PRIVATE in guest_memfd and so in-place conversion is the only option.

Quoted 6 lines

On the other hand, while gmem_in_place_conversion_enabled is better, it's not quite accurate either because userspace isn't strictly required to do in-place conversion.

As for a kvm_ prefix, IMO gmem_ is sufficient for a namespace, and not having kvm_ is consistent with most module params in KVM.

Maybe we should have a helper function instead of accessing the module parameter directly, then the gmem_in_place_conversion could just stay file-local and kvm_gmem_in_place_conversion() would be used by other code that wants to obtain the value.

Instead of the

#define mem_in_place_conversion false

Quoted 7 lines
Quoted 5 lines

We'd have

#define kvm_gmem_in_place_conversion() false

just a thought ...

FWIW, I'd rather prefix kvm_ than add a wrapper to get a boolean.

Could you elaborate on the rationale for a wrapper? I don't really see a reason to encapsulate this boolean since this flag should be global to KVM, it toggles between tracking shared/private status at the VM or guest_memfd level. Since KVM is a module I guess it's already global just to KVM?

On the naming, Sashiko did point out somewhere that gmem_in_place_conversion as a global might end up shadowing or being shadowed by something else.

Prefixing kvm_ was one of Sashiko's suggestions too.

I don't really have a strong opinion on this but I'd like to know what to put in v11.

Will using

module_param_named(gmem_in_place_conversion,
kvm_gmem_in_place_conversion, bool, 0444);

help? The module param doesn't have the kvm prefix but the variable does.

Reviewed-by on v10 7/41
#24 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

In reply to Ackerley Tng
Quoted 8 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

Quoted 5 lines

The names will finalized eventually and it won't impact the implementation. So,

Quoted 113 lines
---
  include/linux/kvm_host.h |  6 ++++++
  virt/kvm/guest_memfd.c   | 26 ++++++++++++++++++++++++++
  virt/kvm/kvm_main.c      | 12 +++++++++++-
  3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +6 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
  #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;
+
  typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
  DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  	return static_call(__kvm_mem_is_private)(kvm, gfn);
  }
  #else
+#define gmem_in_place_conversion false
+
  static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  {
  	return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  #endif /* kvm_arch_has_private_mem */
 
  #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);
+
  int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
  		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
  		     int *max_order);
virt/kvm/guest_memfd.c +26 -0
index 1f1ae61b4878b..75979c885e030 100644
@@ -475,6 +475,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
  	return 0;
  }
 
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+	struct inode *inode;
+
+	if (!slot)
+		return 0;
+
+	CLASS(gmem_get_file, file)(slot);
+	if (!file)
+		return 0;
+
+	inode = file_inode(file);
+
+	/*
+	 * Rely on the maple tree's internal RCU lock to ensure a stable result.
+	 * This result can become stale as soon as the lock is dropped, so the
+	 * caller _must_ protect consumption of private vs. shared either by
+	 * holding guest_memfd's invalidate lock for the entire duration, or by
+	 * checking mmu_invalidate_retry_gfn() under mmu_lock to serialize
+	 * against concurrent attribute updates.
+	 */
+	return kvm_gmem_is_private_mem(inode, kvm_gmem_get_index(slot, gfn));
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private);
+
  static struct file_operations kvm_gmem_fops = {
  	.mmap		= kvm_gmem_mmap,
  	.open		= generic_file_open,
virt/kvm/kvm_main.c +11 -1
index a01d336eab18b..0a5a1bee8de0f 100644
@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);
  static bool __ro_after_init allow_unsafe_mappings;
  module_param(allow_unsafe_mappings, bool, 0444);
 
+#ifdef kvm_arch_has_private_mem
+bool __ro_after_init gmem_in_place_conversion = false;
+#endif
+
  /*
   * Ordering of locks:
   *
@@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
  static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
  {
  #ifdef kvm_arch_has_private_mem
+	if (gmem_in_place_conversion)
+		return 0;
+
  	if (!kvm || kvm_arch_has_private_mem(kvm))
  		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
  #endif
@@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_mem_is_private);
 
  static void kvm_init_memory_attributes(void)
  {
+	if (gmem_in_place_conversion)
+		static_call_update(__kvm_mem_is_private, kvm_gmem_is_private);
  #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
+	else
+		static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
  #endif
  }
  #else
reviewer Reviewed-by on v10 7/41
#25 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

In reply to Ackerley Tng
Quoted 9 lines · On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay…

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

Quoted 5 lines

Cheers,
/fuad

Quoted 117 lines
---
 include/linux/kvm_host.h |  6 ++++++
 virt/kvm/guest_memfd.c   | 26 ++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      | 12 +++++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +6 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;
+
 typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
 DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
        return static_call(__kvm_mem_is_private)(kvm, gfn);
 }
 #else
+#define gmem_in_place_conversion false
+
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
        return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);
+
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
                     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
                     int *max_order);
virt/kvm/guest_memfd.c +26 -0
index 1f1ae61b4878b..75979c885e030 100644
@@ -475,6 +475,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
        return 0;
 }
 
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+       struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+       struct inode *inode;
+
+       if (!slot)
+               return 0;
+
+       CLASS(gmem_get_file, file)(slot);
+       if (!file)
+               return 0;
+
+       inode = file_inode(file);
+
+       /*
+        * Rely on the maple tree's internal RCU lock to ensure a stable result.
+        * This result can become stale as soon as the lock is dropped, so the
+        * caller _must_ protect consumption of private vs. shared either by
+        * holding guest_memfd's invalidate lock for the entire duration, or by
+        * checking mmu_invalidate_retry_gfn() under mmu_lock to serialize
+        * against concurrent attribute updates.
+        */
+       return kvm_gmem_is_private_mem(inode, kvm_gmem_get_index(slot, gfn));
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private);
+
 static struct file_operations kvm_gmem_fops = {
        .mmap           = kvm_gmem_mmap,
        .open           = generic_file_open,
virt/kvm/kvm_main.c +11 -2
index a01d336eab18b..0a5a1bee8de0f 100644
@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);
 static bool __ro_after_init allow_unsafe_mappings;
 module_param(allow_unsafe_mappings, bool, 0444);
 
+#ifdef kvm_arch_has_private_mem
+bool __ro_after_init gmem_in_place_conversion = false;
+#endif
+
 /*
  * Ordering of locks:
  *
@@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 #ifdef kvm_arch_has_private_mem
+       if (gmem_in_place_conversion)
+               return 0;
+
        if (!kvm || kvm_arch_has_private_mem(kvm))
                return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 #endif
@@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_mem_is_private);
 
 static void kvm_init_memory_attributes(void)
 {
+       if (gmem_in_place_conversion)
+               static_call_update(__kvm_mem_is_private, kvm_gmem_is_private);
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-       static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
+       else
+               static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
 #endif
 }
 #else
 
--

2.55.0.654.g21b8a5bc05-goog

#26 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

In reply to Ackerley Tng
Quoted 7 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

I find this patch changes the default memory type for a @gfn.

- When memory attribute is tracked per-VM, the default memory type is always shared.

- when gmem_in_place_conversion is true,
- if the @gfn has no memslot, or the memslot where the @gfn locates doesn't have gmem bound, the default memory type is shared,
- otherwise, the default memory type is determined by the GUEST_MEMFD_FLAG_INIT_SHARED flag.

I think we should document this change.

Quoted 118 lines
---
  include/linux/kvm_host.h |  6 ++++++
  virt/kvm/guest_memfd.c   | 26 ++++++++++++++++++++++++++
  virt/kvm/kvm_main.c      | 12 +++++++++++-
  3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +6 -0
index 65fbce46b63f4..9477ecebbbced 100644
@@ -2580,6 +2580,8 @@ static inline bool kvm_vm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
  #ifdef kvm_arch_has_private_mem
+extern bool gmem_in_place_conversion;
+
  typedef bool (kvm_mem_is_private_t)(struct kvm *kvm, gfn_t gfn);
  DECLARE_STATIC_CALL(__kvm_mem_is_private, kvm_mem_is_private_t);
 
@@ -2588,6 +2590,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  	return static_call(__kvm_mem_is_private)(kvm, gfn);
  }
  #else
+#define gmem_in_place_conversion false
+
  static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  {
  	return false;
@@ -2595,6 +2599,8 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
  #endif /* kvm_arch_has_private_mem */
 
  #ifdef CONFIG_KVM_GUEST_MEMFD
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);
+
  int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
  		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
  		     int *max_order);
virt/kvm/guest_memfd.c +26 -0
index 1f1ae61b4878b..75979c885e030 100644
@@ -475,6 +475,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
  	return 0;
  }
 
+bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+	struct inode *inode;
+
+	if (!slot)
+		return 0;
+
+	CLASS(gmem_get_file, file)(slot);
+	if (!file)
+		return 0;
+
+	inode = file_inode(file);
+
+	/*
+	 * Rely on the maple tree's internal RCU lock to ensure a stable result.
+	 * This result can become stale as soon as the lock is dropped, so the
+	 * caller _must_ protect consumption of private vs. shared either by
+	 * holding guest_memfd's invalidate lock for the entire duration, or by
+	 * checking mmu_invalidate_retry_gfn() under mmu_lock to serialize
+	 * against concurrent attribute updates.
+	 */
+	return kvm_gmem_is_private_mem(inode, kvm_gmem_get_index(slot, gfn));
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private);
+
  static struct file_operations kvm_gmem_fops = {
  	.mmap		= kvm_gmem_mmap,
  	.open		= generic_file_open,
virt/kvm/kvm_main.c +11 -1
index a01d336eab18b..0a5a1bee8de0f 100644
@@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);
  static bool __ro_after_init allow_unsafe_mappings;
  module_param(allow_unsafe_mappings, bool, 0444);
 
+#ifdef kvm_arch_has_private_mem
+bool __ro_after_init gmem_in_place_conversion = false;
+#endif
+
  /*
   * Ordering of locks:
   *
@@ -2423,6 +2427,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
  static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
  {
  #ifdef kvm_arch_has_private_mem
+	if (gmem_in_place_conversion)
+		return 0;
+
  	if (!kvm || kvm_arch_has_private_mem(kvm))
  		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
  #endif
@@ -2634,8 +2641,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_mem_is_private);
 
  static void kvm_init_memory_attributes(void)
  {
+	if (gmem_in_place_conversion)
+		static_call_update(__kvm_mem_is_private, kvm_gmem_is_private);
  #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
+	else
+		static_call_update(__kvm_mem_is_private, kvm_vm_mem_is_private);
  #endif
  }
  #else
#27 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 14 lines · Xiaoyao Li
Quoted 7 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

I find this patch changes the default memory type for a @gfn.

- When memory attribute is tracked per-VM, the default memory type is always shared.

Quoted 8 lines

- when gmem_in_place_conversion is true,
- if the @gfn has no memslot, or the memslot where the @gfn locates doesn't have gmem bound, the default memory type is shared,
- otherwise, the default memory type is determined by the GUEST_MEMFD_FLAG_INIT_SHARED flag.

I think we should document this change.

In a way it's not really a "change" since gmem_in_place_conversion is set up as false in this patch.

When it is enabled in a later patch, another way to see it is that the default remains "shared unless defined as private".

Before:

+ no memslot, or memslot not bound to gmem => shared
+ memory is shared unless VM ioctl used to make gfn private.

After:

+ no memslot, or memslot not bound to gmem => still shared
+ otherwise, ask gmem about status, which I think is already captured in the module param concept.
+ The very usage of gmem (without INIT_SHARED) is defining memory as private, I think that is already documented elsewhere when INIT_SHARED was introduced, that now the default is private.

So in summary, I feel that this has already been documented in various places. I'll also add the following in v11's "KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes", in Documentation/admin-guide/kernel-parameters.txt:

kvm.gmem_in_place_conversion=
[KVM] Controls whether KVM enables in-place conversion support for guest_memfd and tracks the private/shared state of memory per guest_memfd instead of per VM.

If enabled (the default), KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 ioctl on guest_memfd file descriptors and disables the legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for private memory state tracking. Only the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to per-guest_memfd tracking; other attributes remain per-VM.

This parameter toggles KVM's in-place conversion capability support. Whether a VMM uses separate backends or out-of-place memory management is determined by userspace VMM design.

Note, this parameter is only available when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place conversion is unconditionally enabled.

Default is Y (on).

Quoted 3 lines
Quoted 3 lines

[...snip...]

#28 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 22 lines · Ackerley Tng
Quoted 20 lines · Xiaoyao Li
Quoted 7 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

I find this patch changes the default memory type for a @gfn.

- When memory attribute is tracked per-VM, the default memory type is always shared.

- when gmem_in_place_conversion is true,
- if the @gfn has no memslot, or the memslot where the @gfn locates doesn't have gmem bound, the default memory type is shared,
- otherwise, the default memory type is determined by the GUEST_MEMFD_FLAG_INIT_SHARED flag.

Quoted 5 lines
Quoted 2 lines

I think we should document this change.

In a way it's not really a "change" since gmem_in_place_conversion is set up as false in this patch.

I agree. It is not this patch making the change but the patch allows gmem_in_place_conversion to be true.

Quoted 22 lines

When it is enabled in a later patch, another way to see it is that the default remains "shared unless defined as private".

Before:

+ no memslot, or memslot not bound to gmem => shared
+ memory is shared unless VM ioctl used to make gfn private.

After:

+ no memslot, or memslot not bound to gmem => still shared
+ otherwise, ask gmem about status, which I think is already captured in the module param concept.
+ The very usage of gmem (without INIT_SHARED) is defining memory as private, I think that is already documented elsewhere when INIT_SHARED was introduced, that now the default is private.

So in summary, I feel that this has already been documented in various places. I'll also add the following in v11's "KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes", in Documentation/admin-guide/kernel-parameters.txt:

Quoted 6 lines

kvm.gmem_in_place_conversion=
[KVM] Controls whether KVM enables in-place conversion support for guest_memfd and tracks the private/shared state of memory per guest_memfd instead of per VM.

If enabled (the default), KVM enables the

I think the default is disabled?

Quoted 9 lines

KVM_SET_MEMORY_ATTRIBUTES2 ioctl on guest_memfd file descriptors and disables the legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for private memory state tracking. Only the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to per-guest_memfd tracking; other attributes remain per-VM.

This parameter toggles KVM's in-place conversion capability support.

I start to think that the term "in-place conversion" seems to read inaccurate. I think it is describing the shared/private conversion of a gfn, and in-place means when a gfn is converted between shared/private, the backend comes from the same gmem page, thus in-place. But KVM doesn't enforce the "in-place".

If "in-place conversion" describes the shared/private conversion of a gmem page, then "in-place" is redundant because the conversion a specific gmem page is always in-place.

Quoted 10 lines

Whether a VMM uses separate backends
or out-of-place memory management is determined by userspace VMM design.

Note, this parameter is only available when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place conversion is unconditionally enabled.

Default is Y (on).

I'm looking at the doc of KVM_SET_USER_MEMORY_REGION2, which reads

# When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes

# userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
# state.  At VM creation time, all memory is shared, i.e. the PRIVATE attribute
# is '0' for all gfns.  Userspace can control whether memory is shared/private by
# toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.
#29 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 54 lines · Xiaoyao Li
Quoted 22 lines · Ackerley Tng
Quoted 20 lines · Xiaoyao Li
Quoted 7 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_mem_is_private() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory.

I find this patch changes the default memory type for a @gfn.

- When memory attribute is tracked per-VM, the default memory type is always shared.

- when gmem_in_place_conversion is true,
- if the @gfn has no memslot, or the memslot where the @gfn locates doesn't have gmem bound, the default memory type is shared,
- otherwise, the default memory type is determined by the GUEST_MEMFD_FLAG_INIT_SHARED flag.

Quoted 5 lines
Quoted 2 lines

I think we should document this change.

In a way it's not really a "change" since gmem_in_place_conversion is set up as false in this patch.

I agree. It is not this patch making the change but the patch allows gmem_in_place_conversion to be true.

Quoted 22 lines

When it is enabled in a later patch, another way to see it is that the default remains "shared unless defined as private".

Before:

+ no memslot, or memslot not bound to gmem => shared
+ memory is shared unless VM ioctl used to make gfn private.

After:

+ no memslot, or memslot not bound to gmem => still shared
+ otherwise, ask gmem about status, which I think is already captured in the module param concept.
+ The very usage of gmem (without INIT_SHARED) is defining memory as private, I think that is already documented elsewhere when INIT_SHARED was introduced, that now the default is private.

So in summary, I feel that this has already been documented in various places. I'll also add the following in v11's "KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes", in Documentation/admin-guide/kernel-parameters.txt:

Quoted 9 lines
Quoted 6 lines

kvm.gmem_in_place_conversion=
[KVM] Controls whether KVM enables in-place conversion support for guest_memfd and tracks the private/shared state of memory per guest_memfd instead of per VM.

If enabled (the default), KVM enables the

I think the default is disabled?

Ah. I'm thinking of the state in the way that we're deprecating non-in-place-conversion. I'll drop "(the default)", since the default depends on the CONFIG flag.

Quoted 16 lines
Quoted 9 lines

KVM_SET_MEMORY_ATTRIBUTES2 ioctl on guest_memfd file descriptors and disables the legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for private memory state tracking. Only the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to per-guest_memfd tracking; other attributes remain per-VM.

This parameter toggles KVM's in-place conversion capability support.

I start to think that the term "in-place conversion" seems to read inaccurate. I think it is describing the shared/private conversion of a gfn, and in-place means when a gfn is converted between shared/private, the backend comes from the same gmem page, thus in-place. But KVM doesn't enforce the "in-place".

Quoted 4 lines

If "in-place conversion" describes the shared/private conversion of a gmem page, then "in-place" is redundant because the conversion a specific gmem page is always in-place.

Hmm, a few people have raised something related to this gmem_in_place_conversion module param's naming: Xiaoyao, Yan, David, and Sean's response is generally that it is confusing, but can't find a better way out. The main consideration around module param naming is that it should be named for the benefit of the admin. We want some name that admins can understand at a high level (for some definition of "high level") what this does.

Do you have a proposal to resolve your concern, considering naming, documentation, comments, code, etc?

Quoted 15 lines
Quoted 10 lines

Whether a VMM uses separate backends
or out-of-place memory management is determined by userspace VMM design.

Note, this parameter is only available when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place conversion is unconditionally enabled.

Default is Y (on).

I'm looking at the doc of KVM_SET_USER_MEMORY_REGION2, which reads

# When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes

Quoted 8 lines
# userspace_addr vs. guest_memfd, based on the gfn's
KVM_MEMORY_ATTRIBUTE_PRIVATE
# state.  At VM creation time, all memory is shared, i.e. the PRIVATE
attribute
# is '0' for all gfns.  Userspace can control whether memory is
shared/private by
# toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as
needed.

I'm not sure how this snippet from the documentation connects with what you'd like changed. Let me know your suggested changes across naming/docs/comments/code to address your concern :)

#30 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 21 lines · Ackerley Tng
Quoted 16 lines · Xiaoyao Li
Quoted 9 lines

KVM_SET_MEMORY_ATTRIBUTES2 ioctl on guest_memfd file descriptors and disables the legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for private memory state tracking. Only the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to per-guest_memfd tracking; other attributes remain per-VM.

This parameter toggles KVM's in-place conversion capability support.

I start to think that the term "in-place conversion" seems to read inaccurate. I think it is describing the shared/private conversion of a gfn, and in-place means when a gfn is converted between shared/private, the backend comes from the same gmem page, thus in-place. But KVM doesn't enforce the "in-place".

Quoted 3 lines

If "in-place conversion" describes the shared/private conversion of a gmem page, then "in-place" is redundant because the conversion a specific gmem page is always in-place.

Quoted 7 lines

Hmm, a few people have raised something related to this gmem_in_place_conversion module param's naming: Xiaoyao, Yan, David, and Sean's response is generally that it is confusing, but can't find a better way out. The main consideration around module param naming is that it should be named for the benefit of the admin. We want some name that admins can understand at a high level (for some definition of "high level") what this does.

Executive decision: use gmem_in_place_conversion. I hear (and largely agree with) the complaints that it's imperfect, but I don't think it's feasible to find a name that can perfectly describe the nuances while still being somewhat succint and intuitive. I.e. gmem_in_place_conversion isn't perfect, but everything else I've seen is much worse.

I'll make sure to call out that gmem_in_place_conversion is imperfect in the pull request, to give Paolo a chance to veto my executive decision.

Quoted 16 lines

Do you have a proposal to resolve your concern, considering naming, documentation, comments, code, etc?

Quoted 13 lines
Quoted 10 lines

Whether a VMM uses separate backends
or out-of-place memory management is determined by userspace VMM design.

Note, this parameter is only available when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place conversion is unconditionally enabled.

Default is Y (on).

I'm looking at the doc of KVM_SET_USER_MEMORY_REGION2, which reads

Quoted 8 lines
Quoted 5 lines

# When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes

# userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
# state.  At VM creation time, all memory is shared, i.e. the PRIVATE attribute
# is '0' for all gfns.  Userspace can control whether memory is shared/private by
# toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.

I'm not sure how this snippet from the documentation connects with what you'd like changed.

It's flat out wrong once in-place conversion lands, because it assumes PRIVATE is tracked per-VM. Something like this?

Documentation/virt/kvm/api.rst +5 -2
index 4eb7e75a7473..c9769e5e7329 100644
@@ -6383,9 +6383,12 @@ on-demand.
 
 When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes
 userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
-state.  At VM creation time, all memory is shared, i.e. the PRIVATE attribute
-is '0' for all gfns.  Userspace can control whether memory is shared/private by
+state.  If in-place conversion is disabled, i.e. PRIVATE is tracked per-VM,
+then at VM creation time, all memory is shared, i.e. the PRIVATE attribute is
+'0' for all gfns.  Userspace can control whether memory is shared/private by
 toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.
+If in-place conversion is enabled, then the starting PRIVATE vs. SHARED state
+of a gfn is determined by the relevant guest_memfd instance.
 
 S390:
 ^^^^^
#31 lore raw

Re: [PATCH v10 07/41] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion

Quoted 27 lines · Sean Christopherson

[...snip...]

Quoted 11 lines
Quoted 8 lines

I'm looking at the doc of KVM_SET_USER_MEMORY_REGION2, which reads

# When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes

# userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
# state.  At VM creation time, all memory is shared, i.e. the PRIVATE attribute
# is '0' for all gfns.  Userspace can control whether memory is shared/private by
# toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.

I'm not sure how this snippet from the documentation connects with what you'd like changed.

It's flat out wrong once in-place conversion lands, because it assumes PRIVATE is tracked per-VM. Something like this?

Documentation/virt/kvm/api.rst +0 -1
index 4eb7e75a7473..c9769e5e7329 100644
@@ -6383,9 +6383,12 @@ on-demand.
 
 When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes
 userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
-state.  At VM creation time, all memory is shared, i.e. the PRIVATE attribute

Documentation/virt/kvm/api.rst on-demand. · line 6394

Quoted 10 lines
Documentation/virt/kvm/api.rst +5 -1
-is '0' for all gfns.  Userspace can control whether memory is shared/private by
+state.  If in-place conversion is disabled, i.e. PRIVATE is tracked per-VM,
+then at VM creation time, all memory is shared, i.e. the PRIVATE attribute is
+'0' for all gfns.  Userspace can control whether memory is shared/private by
 toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.
+If in-place conversion is enabled, then the starting PRIVATE vs. SHARED state
+of a gfn is determined by the relevant guest_memfd instance.
 
 S390:
 ^^^^^

Will apply this, thanks!

#32 lore raw

PATCH v10 8/41 KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Move the kvm_arch_has_private_mem() stub and a few guest_memfd function definitions/declarations "down" in kvm_host.h to utilize existing #ifdefs, and so that related code is clustered together.

No functional change intended.

---
 include/linux/kvm_host.h | 37 ++++++++++++++++---------------------
 1 file changed, 16 insertions(+), 21 deletions(-)
include/linux/kvm_host.h +16 -21
index 9477ecebbbced..dfdb3c97a7ac9 100644
@@ -722,27 +722,6 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 }
 #endif
 
-#ifndef kvm_arch_has_private_mem
-static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
-{
-	return false;
-}
-#endif
-
-#ifdef CONFIG_KVM_GUEST_MEMFD
-bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
-
-static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
-{
-	u64 flags = GUEST_MEMFD_FLAG_MMAP;
-
-	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
-		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
-
-	return flags;
-}
-#endif
-
 #ifndef kvm_arch_has_readonly_mem
 static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm)
 {
@@ -2592,6 +2571,11 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 #else
 #define gmem_in_place_conversion false
 
+static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return false;
+}
+
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
@@ -2600,6 +2584,17 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 bool kvm_gmem_is_private(struct kvm *kvm, gfn_t gfn);
+bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
+
+static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
+{
+	u64 flags = GUEST_MEMFD_FLAG_MMAP;
+
+	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
+		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
+
+	return flags;
+}
 
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
maintainer Reviewed-by on v10 8/41
#33 lore raw

Re: [PATCH v10 08/41] KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h

Quoted 8 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Move the kvm_arch_has_private_mem() stub and a few guest_memfd function definitions/declarations "down" in kvm_host.h to utilize existing #ifdefs, and so that related code is clustered together.

No functional change intended.

Quoted 7 lines

---

Signature · 3 lines
Cheers,

David
#34 lore raw

PATCH v10 9/41 KVM: guest_memfd: Filter both shared and private when invalidating

In reply to Ackerley Tng

Before conversion, a guest_memfd could be either all shared, or all private, configured at creation time using the INIT_SHARED flag. Hence, when zapping pages from stage 2 page tables, guest_memfd can filter which to zap based on the initial shared/private state.

With conversion, guest_memfd tracks shared/private state on a per-page level, so a range can contain both private and shared pages. Zap both private and shared pages for simplicity.

An alternative would be to iterate guest_memfd attributes and only zap both if shared and private pages exist within the range. Setting both the shared and private filters lets the zapping logic do that iteration instead.

---
 virt/kvm/guest_memfd.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)
virt/kvm/guest_memfd.c +8 -16
index 75979c885e030..b1537226e5adf 100644
@@ -136,14 +136,6 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	return folio;
 }
 
-static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
-{
-	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
-		return KVM_FILTER_SHARED;
-
-	return KVM_FILTER_PRIVATE;
-}
-
 static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 					pgoff_t end,
 					enum kvm_gfn_range_filter attr_filter)
@@ -186,15 +178,13 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 }
 
 static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
-				      pgoff_t end)
+				      pgoff_t end,
+				      enum kvm_gfn_range_filter filter)
 {
-	enum kvm_gfn_range_filter attr_filter;
 	struct gmem_file *f;
 
-	attr_filter = kvm_gmem_get_invalidate_filter(inode);
-
 	kvm_gmem_for_each_file(f, inode)
-		__kvm_gmem_invalidate_start(f, start, end, attr_filter);
+		__kvm_gmem_invalidate_start(f, start, end, filter);
 }
 
 static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
@@ -229,7 +219,8 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
 	 */
 	filemap_invalidate_lock(inode->i_mapping);
 
-	kvm_gmem_invalidate_start(inode, start, end);
+	kvm_gmem_invalidate_start(inode, start, end,
+				  KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
 
 	truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
 
@@ -344,7 +335,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
 	__kvm_gmem_invalidate_start(f, 0, -1ul,
-				    kvm_gmem_get_invalidate_filter(inode));
+				    KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
 	__kvm_gmem_invalidate_end(f, 0, -1ul);
 
 	list_del(&f->entry);
@@ -525,7 +516,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	start = folio->index;
 	end = start + folio_nr_pages(folio);
 
-	kvm_gmem_invalidate_start(mapping->host, start, end);
+	kvm_gmem_invalidate_start(mapping->host, start, end,
+				  KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
 
 	/*
 	 * Do not truncate the range, what action is taken in response to the
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#35 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 3 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Before conversion, a guest_memfd could be either all shared, or all

You mean "in-place conversion support" ?

Quoted 5 lines

private, configured at creation time using the INIT_SHARED flag. Hence, when zapping pages from stage 2 page tables, guest_memfd can filter which to zap based on the initial shared/private state.

With conversion, guest_memfd tracks shared/private state on a per-page

Same here.

Quoted 6 lines

level, so a range can contain both private and shared pages. Zap both private and shared pages for simplicity.

An alternative would be to iterate guest_memfd attributes and only zap both if shared and private pages exist within the range. Setting both the shared and private filters lets the zapping logic do that iteration instead.

I mean, we just want to zap anything that belongs to guest_memfd, independent of shared vs. private, really?

IOW, it's not about shared vs. private, but really about zapping anything that belongs to guest_memfd.

So, couldn't there just be a KVM_FILTER_GMEM thingy instead?

But I didn't quite digest how these filters are used.

I can see that kvm_gfn_range_filter_to_root_types() does some magic to them, but I am not a KVM MMU expert to know what KVM_MIRROR_ROOTS would mean.

Signature · 3 lines
Cheers,

David
#36 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Before conversion, a guest_memfd could be either all shared, or all

You mean "in-place conversion support" ?

Not necessarily. This is supposed to be :

"At creation, ...

Quoted 16 lines
Quoted 6 lines

private, configured at creation time using the INIT_SHARED flag. Hence, when zapping pages from stage 2 page tables, guest_memfd can filter which to zap based on the initial shared/private state.

With conversion, guest_memfd tracks shared/private state on a per-page

Same here.

Quoted 6 lines

level, so a range can contain both private and shared pages. Zap both private and shared pages for simplicity.

An alternative would be to iterate guest_memfd attributes and only zap both if shared and private pages exist within the range. Setting both the shared and private filters lets the zapping logic do that iteration instead.

Quoted 5 lines

I mean, we just want to zap anything that belongs to guest_memfd, independent of shared vs. private, really?

IOW, it's not about shared vs. private, but really about zapping anything that belongs to guest_memfd.

Correct. But we want to control what we "zap" (invalidate range) when we do the "conversion". e.g., if some ranges are already private, we don't want to zap those pages, as they might have some data "populated" by the VMM. For more context:

https://lore.kernel.org/all/[email protected]

Suzuki

Quoted 8 lines

So, couldn't there just be a KVM_FILTER_GMEM thingy instead?

But I didn't quite digest how these filters are used.

I can see that kvm_gfn_range_filter_to_root_types() does some magic to them, but I am not a KVM MMU expert to know what KVM_MIRROR_ROOTS would mean.

#37 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 4 lines · Suzuki K Poulose
Quoted 3 lines · David Hildenbrand (Arm)
Quoted 2 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Before conversion, a guest_memfd could be either all shared, or all

You mean "in-place conversion support" ?

Not necessarily. This is supposed to be :

"At creation, ...

I've been thinking of this series as the conversions series hence this description, David is right about being more specific with "in-place conversion support".

I didn't notice that it could also be understood as "at creation" vs later on in the guest_memfd lifecycle. I did intend it to be about in-place conversion support, since I was trying to justify why we need to zap both shared and private pages instead of zapping based on the INIT_SHARED flag.

Quoted 22 lines
Quoted 16 lines
Quoted 6 lines

private, configured at creation time using the INIT_SHARED flag. Hence, when zapping pages from stage 2 page tables, guest_memfd can filter which to zap based on the initial shared/private state.

With conversion, guest_memfd tracks shared/private state on a per-page

Same here.

Quoted 6 lines

level, so a range can contain both private and shared pages. Zap both private and shared pages for simplicity.

An alternative would be to iterate guest_memfd attributes and only zap both if shared and private pages exist within the range. Setting both the shared and private filters lets the zapping logic do that iteration instead.

Quoted 5 lines

I mean, we just want to zap anything that belongs to guest_memfd, independent of shared vs. private, really?

IOW, it's not about shared vs. private, but really about zapping anything that belongs to guest_memfd.

Quoted 10 lines

Correct. But we want to control what we "zap" (invalidate range) when we do the "conversion". e.g., if some ranges are already private, we don't want to zap those pages, as they might have some data "populated" by the VMM. For more context:

https://lore.kernel.org/all/[email protected]

Suzuki

Looking again, there are actually 2 changes in this patch,

1. Zap both shared and private memory.
2. Allow zap requester to request what to zap by providing a filter.

(2) is preparation for in-place conversions, where Suzuki pointed out that in the conversion process introduced with in-place conversions, guest_memfd should only request to zap memory that is not aligned with the requested outcome of the conversion in terms of shared/private state, to avoid over-zapping.

e.g. for shared to private conversions, only zap shared memory, zapping private might destroy data.

How about this, I can split this patch as follows:

1. Update this patch to just do (2)

Commit subject: Allow zapping requester to provide range filter

In the conversion process introduced with in-place conversions, guest_memfd should only request to zap memory that is not aligned with the requested outcome of the conversion in terms of shared/private state, to avoid over-zapping.

Prepare for that by allowing zapping requester to provide range filter so the requester can zap private, shared or both types of pages.

2. In the next patch "Add base support for KVM_SET_MEMORY_ATTRIBUTES2", call kvm_gmem_invalidate_start() for conversions providing just KVM_FILTER_PRIVATE or KVM_FILTER_SHARED - no change to the next patch, mostly.

3. Add a simplification patch to always zap both PRIVATE and SHARED for the truncation, inode release and memory failure calls to kvm_gmem_invalidate_start()

With guest_memfd in-place conversions, guest_memfd can iterate the maple tree to look up whether to zap PRIVATE, SHARED, or both based on the range being zapped, but I think that's doing extra work if the goal is to zap everything from guest_memfd.

Before gmem it was an easy lookup to check the guest_memfd init time flag, but with conversions it requires iteration, and truncation/release/memory failure aren't particularly time sensitive.

Sean, do you know if looking up attributes in gmem to feed the KVM MMU the smallest set of pages to zap will improve performance significantly? Or if there's any other reason to do this lookup (more complexity in gmem)?

Quoted 8 lines
Quoted 8 lines

So, couldn't there just be a KVM_FILTER_GMEM thingy instead?

But I didn't quite digest how these filters are used.

I can see that kvm_gfn_range_filter_to_root_types() does some magic to them, but I am not a KVM MMU expert to know what KVM_MIRROR_ROOTS would mean.

#38 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 4 lines · Ackerley Tng

Sean, do you know if looking up attributes in gmem to feed the KVM MMU the smallest set of pages to zap will improve performance significantly? Or if there's any other reason to do this lookup (more complexity in gmem)?

While working through this with Ackerley, I realized this patch is buggy. When in-place conversion is NOT supported, then as evidenced by the current code, invalidations are guaranteed to only affect one of SHARED vs. PRIVATE. And if we change that to zap both, we risk overzapping. I.e. it's not just the cost of the extra MMU walk, it could also be a functional bug.

Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

That latter case in particular would be a functional bug, as spuriously zapping PRIVATE SPTEs is fatal to TDX (destroys the memory contents).

So, for the initial change, we want this (full patch below):

virt/kvm/guest_memfd.c +3 -0
index 75979c885e03..8ff2ec148614 100644
@@ -138,6 +138,9 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 
 static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
 {
+	if (gmem_in_place_conversion)
+		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
+
 	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
 		return KVM_FILTER_SHARED;
 

And then in the main in-place conversion patch, have the conversion flow to only zap tap the "previous" types (with prep work as needed). Ideally, that would be done *after* the main conversion patch, i.e. as an optimization, so that we get a nice bisection point if it's somehow buggy. However, Ackerley pointed out that the conversion flow invalidates the entire range if the attributes of any gfn within the range is changing. Addressing that would be rather annoying, e.g. there would need to be multiple invalidation ranges to deal with interpolated conversions, so going straight to a "zap only the previous" is probably the least awful option.

---
From: Sean Christopherson <[email protected]>
Date: Wed, 19 Aug 2026 18:06:38 -0700
Subject: [PATCH] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings
for in-place conversions

When removing one or more folios from a guest_memfd instance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.

Opportunistically rename the helper to capture that it returns the a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).

Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings of a different gmem instance. The latter case in particular would be a functional bug, as spuriously zapping PRIVATE SPTEs is fatal to TDX, as doing so destroys the contents of the memory.

---
 virt/kvm/guest_memfd.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
virt/kvm/guest_memfd.c +6 -5
index 75979c885e03..7ae05ce3cd14 100644
@@ -136,8 +136,11 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	return folio;
 }
 
-static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
+static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
 {
+	if (gmem_in_place_conversion)
+		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
+
 	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
 		return KVM_FILTER_SHARED;
 
@@ -188,11 +191,9 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
 				      pgoff_t end)
 {
-	enum kvm_gfn_range_filter attr_filter;
+	enum kvm_gfn_range_filter attr_filter  = kvm_gmem_get_all_gfns_filter(inode);
 	struct gmem_file *f;
 
-	attr_filter = kvm_gmem_get_invalidate_filter(inode);
-
 	kvm_gmem_for_each_file(f, inode)
 		__kvm_gmem_invalidate_start(f, start, end, attr_filter);
 }
@@ -344,7 +345,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
 	__kvm_gmem_invalidate_start(f, 0, -1ul,
-				    kvm_gmem_get_invalidate_filter(inode));
+				    kvm_gmem_get_all_gfns_filter(inode));
 	__kvm_gmem_invalidate_end(f, 0, -1ul);
 
 	list_del(&f->entry);

base-commit: 620f8362aaec3848ee8f465949a0701175fe0896 --

#39 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 87 lines · Sean Christopherson
Quoted 4 lines · Ackerley Tng

Sean, do you know if looking up attributes in gmem to feed the KVM MMU the smallest set of pages to zap will improve performance significantly? Or if there's any other reason to do this lookup (more complexity in gmem)?

While working through this with Ackerley, I realized this patch is buggy. When in-place conversion is NOT supported, then as evidenced by the current code, invalidations are guaranteed to only affect one of SHARED vs. PRIVATE. And if we change that to zap both, we risk overzapping. I.e. it's not just the cost of the extra MMU walk, it could also be a functional bug.

Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

That latter case in particular would be a functional bug, as spuriously zapping PRIVATE SPTEs is fatal to TDX (destroys the memory contents).

So, for the initial change, we want this (full patch below):

virt/kvm/guest_memfd.c +3 -0
index 75979c885e03..8ff2ec148614 100644
@@ -138,6 +138,9 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 
  static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
  {
+	if (gmem_in_place_conversion)
+		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
+
  	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
  		return KVM_FILTER_SHARED;
 

And then in the main in-place conversion patch, have the conversion flow to only zap tap the "previous" types (with prep work as needed). Ideally, that would be done *after* the main conversion patch, i.e. as an optimization, so that we get a nice bisection point if it's somehow buggy. However, Ackerley pointed out that the conversion flow invalidates the entire range if the attributes of any gfn within the range is changing. Addressing that would be rather annoying, e.g. there would need to be multiple invalidation ranges to deal with interpolated conversions, so going straight to a "zap only the previous" is probably the least awful option.

---
From: Sean Christopherson <[email protected]>
Date: Wed, 19 Aug 2026 18:06:38 -0700
Subject: [PATCH] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings
for in-place conversions

When removing one or more folios from a guest_memfd instance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped.

Opportunistically rename the helper to capture that it returns the a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place).

Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings of a different gmem instance. The latter case in particular would be a functional bug, as spuriously zapping PRIVATE SPTEs is fatal to TDX, as doing so destroys the contents of the memory.

---
  virt/kvm/guest_memfd.c | 11 ++++++-----
  1 file changed, 6 insertions(+), 5 deletions(-)
virt/kvm/guest_memfd.c +0 -0
index 75979c885e03..7ae05ce3cd14 100644

virt/kvm/guest_memfd.c static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) · line 142

Quoted 8 lines
virt/kvm/guest_memfd.c +3 -1
@@ -136,8 +136,11 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
  	return folio;
  }
 
-static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
+static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
  {
+	if (gmem_in_place_conversion)
+		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;

If I understand correctly, above diff is dead code and will change according to

And then in the main in-place conversion patch, have the conversion flow to only zap tap the "previous" types (with prep work as needed).

If so, why bother adding the change?

Quoted 28 lines
virt/kvm/guest_memfd.c +2 -4
  	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
  		return KVM_FILTER_SHARED;
 
@@ -188,11 +191,9 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
  static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
  				      pgoff_t end)
  {
-	enum kvm_gfn_range_filter attr_filter;
+	enum kvm_gfn_range_filter attr_filter  = kvm_gmem_get_all_gfns_filter(inode);
  	struct gmem_file *f;
 
-	attr_filter = kvm_gmem_get_invalidate_filter(inode);
-
  	kvm_gmem_for_each_file(f, inode)
  		__kvm_gmem_invalidate_start(f, start, end, attr_filter);
  }
@@ -344,7 +345,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
  	 * memory, as its lifetime is associated with the inode, not the file.
  	 */
  	__kvm_gmem_invalidate_start(f, 0, -1ul,
-				    kvm_gmem_get_invalidate_filter(inode));
+				    kvm_gmem_get_all_gfns_filter(inode));
  	__kvm_gmem_invalidate_end(f, 0, -1ul);
 
  	list_del(&f->entry);

base-commit: 620f8362aaec3848ee8f465949a0701175fe0896 --

#40 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 22 lines · Xiaoyao Li
Quoted 21 lines · Sean Christopherson
Quoted 4 lines · Ackerley Tng

Sean, do you know if looking up attributes in gmem to feed the KVM MMU the smallest set of pages to zap will improve performance significantly? Or if there's any other reason to do this lookup (more complexity in gmem)?

While working through this with Ackerley, I realized this patch is buggy. When in-place conversion is NOT supported, then as evidenced by the current code, invalidations are guaranteed to only affect one of SHARED vs. PRIVATE. And if we change that to zap both, we risk overzapping. I.e. it's not just the cost of the extra MMU walk, it could also be a functional bug.

Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

That latter case in particular would be a functional bug, as spuriously zapping PRIVATE SPTEs is fatal to TDX (destroys the memory contents).

...

Quoted 3 lines
Quoted 3 lines

Opportunistically rename the helper to capture that it returns the a filter for all gfns in anticipation of zapping only the previous mapping types on conversion.

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 11 lines

virt/kvm/guest_memfd.c static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) · line 142

Quoted 5 lines
virt/kvm/guest_memfd.c +3 -1
-static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
+static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
  {
+	if (gmem_in_place_conversion)
+		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;

If I understand correctly, above diff is dead code and will change according to

And then in the main in-place conversion patch, have the conversion flow to only zap tap the "previous" types (with prep work as needed).

No, my thought is to keep the newly named kvm_gmem_get_all_gfns_filter() as-is, continue using it flows where KVM needs to zap everything, e.g. PUNCH_HOLE, and have the to-PRIVATE conversion flow open code KVM_FILTER_SHARED.

#41 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

In reply to Sean Christopherson
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

#42 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 10 lines · Xiaoyao Li
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

I think this is for when userspace uses an mmap'able guest_memfd instance to handle shared memory, and a 'normal' guest_memfd instance for private memory. Each instance is bound to the same memslot/GPA range, and KVM_SET_MEMORY_ATTRIBUTES handles switching between the 2.

-Mike

#43 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 8 lines · Michael Roth
Quoted 7 lines · Xiaoyao Li
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

Quoted 8 lines
Quoted 3 lines

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

I think this is for when userspace uses an mmap'able guest_memfd instance to handle shared memory, and a 'normal' guest_memfd instance for private memory. Each instance is bound to the same memslot/GPA range, and KVM_SET_MEMORY_ATTRIBUTES handles switching between the 2.

Yep, exactly.

#44 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Michael Roth
Quoted 7 lines · Xiaoyao Li
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

I think this is for when userspace uses an mmap'able guest_memfd instance to handle shared memory, and a 'normal' guest_memfd instance for private memory. Each instance is bound to the same memslot/GPA range, and KVM_SET_MEMORY_ATTRIBUTES handles switching between the 2.

What I didn't figure out is exactly how the two gmem instances are bound to the same memslot.

The case I can imagine is

1. create gmem1 with GUEST_MEMFD_FLAG_MMAP and GUEST_MEMFD_FLAG_INIT_SHARED, and get fd1. mmap the returned fd1 to get a hva.

2. create gmem2 to get a fd2.

3. call KVM_SET_USER_MEMORY_REGION2 with KVM_MEM_GUEST_MEMFD flag. Pass the @hva from 1) to 'userspace_addr' field and pass the gmem fd2 to 'guest_memfd' field.

However, with this case, only gmem2 is bound to the memslot while gmem1 is not. Following PUNCH_HOLE on gmem1 doesn't invalidate any mappings because the f->bindings of it is empty.

Do I miss anything?

#45 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 34 lines · Xiaoyao Li
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Michael Roth
Quoted 7 lines · Xiaoyao Li
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

I think this is for when userspace uses an mmap'able guest_memfd instance to handle shared memory, and a 'normal' guest_memfd instance for private memory. Each instance is bound to the same memslot/GPA range, and KVM_SET_MEMORY_ATTRIBUTES handles switching between the 2.

What I didn't figure out is exactly how the two gmem instances are bound to the same memslot.

The case I can imagine is

1. create gmem1 with GUEST_MEMFD_FLAG_MMAP and GUEST_MEMFD_FLAG_INIT_SHARED, and get fd1. mmap the returned fd1 to get a hva.

2. create gmem2 to get a fd2.

3. call KVM_SET_USER_MEMORY_REGION2 with KVM_MEM_GUEST_MEMFD flag. Pass the @hva from 1) to 'userspace_addr' field and pass the gmem fd2 to 'guest_memfd' field.

Quoted 5 lines

However, with this case, only gmem2 is bound to the memslot while gmem1 is not. Following PUNCH_HOLE on gmem1 doesn't invalidate any mappings because the f->bindings of it is empty.

Do I miss anything?

I think Xiaoyao is right about this, the "vice versa" part is wrong, since gmem1 has no bindings. Even after kvm_gmem_invalidate_start(), going via MMU notifiers, that invalidation invalidates only KVM_FILTER_SHARED, so only gmem1's mappings will get invalidated.

The earlier sentence about PUNCH_HOLE on the PRIVATE guest_memfd (gmem2) is still correct.

Here's a repeat of the above in pseudocode. (I wanted to make gmem2's flags=0 more explicit.)

gmem1 = KVM_CREATE_GUEST_MEMFD(flags=MMAP | INIT_SHARED) gmem2 = KVM_CREATE_GUEST_MEMFD(flags=0) userspace_addr = mmap(gmem1) KVM_SET_USER_MEMORY_REGION2(userspace_addr, gmem2)

gmem2 has no MMAP flag, so that KVM will actually fault from gmem1's memory.

Walking through PUNCH_HOLE on gmem2 again:

gmem_in_place_conversion=n:
* conversions go to the VM, shared/private tracked at VM level.
stage 2 page tables:
* private mappings from gmem2
* shared mappings from gmem1

If guest_memfd invalidates KVM_FILTER_PRIVATE | KVM_FILTER_SHARED, then gmem1's mappings in the stage2 page tables (which are shared) will be invalidated along with gmem2's private mappings. That's not great since we only really wanted to invalidate gmem2's mappings.

If guest_memfd invalidates only KVM_FILTER_PRIVATE (since gmem2 was created with flags=0), so we will always invalidate just the private mappings, which is what we expect.

Now I think there's an issue here: with gmem_in_place_conversion=y, everything is still the same as above, other than where shared/private is tracked.

gmem_in_place_conversion=y:
* conversions go to gmem, shared/private tracked in gmem2.
stage 2 page tables:
* private mappings from gmem2
* shared mappings from gmem1

This time, the filter lookup will return KVM_FILTER_PRIVATE | KVM_FILTER_SHARED and we will still invalidate memory from gmem1.

In conversions, a to-private conversion on gmem2 will invalidate KVM_FILTER_SHARED, which also invalidates gmem1's memory.

Is this just a case of user error? That with gmem_in_place_conversion, userspace should not use userspace_addr from something other than the gmem for the same memslot?

We previously discussed that validating userspace_addr belonging to the gmem for the same memslot is pointless since userspace can later remap userspace_addr to something else.

One option is to say if the gmem doesn't have MMAP set, always only invalidate (filter &= KVM_FILTER_PRIVATE). That means a to-private conversion on gmem2 will not invalidate the shared mappings from gmem1. That would break implicit conversions to shared, since a to-shared access sees the shared mappings and doesn't cause an exit.

Perhaps we can iterate the maple tree and present folios and invalidate exactly the folios matching the shared/private status and gmem, but this is quite complicated and I hope we can avoid this.

What if we say, for gmem_in_place_conversion=y, set KVM_MEMSLOT_GMEM_ONLY for all guest_memfd memslots. This way, the host can still use gmem1 for userspace_addr (user error).

KVM won't be using userspace_addr at all other than kvm_read_guest / kvm_write_guest, and for those, we previously discussed that it is a user error.

Xiaoyao, any suggestions to solve the problem?

Meanwhile, I'll still post v11 to resolve the other comments first.

#46 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 36 lines · Ackerley Tng
Quoted 34 lines · Xiaoyao Li
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Michael Roth
Quoted 7 lines · Xiaoyao Li
Quoted 3 lines · Sean Christopherson

And vice versa, a PUNCH_HOLE on a SHARED gmem (if userspace is
using an INIT_SHARED gmem for the shared branch of a memslot) could invalidate the PRIVATE mappings (of a different gmem instance).

I'm wondering now how this could happen.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

the requirement for PUNCH_HOLE on gmem to trigger mapping invalidation is the gmem is bound with the memslot. But how can a memslot bound with two gmem instances?

I think this is for when userspace uses an mmap'able guest_memfd instance to handle shared memory, and a 'normal' guest_memfd instance for private memory. Each instance is bound to the same memslot/GPA range, and KVM_SET_MEMORY_ATTRIBUTES handles switching between the 2.

What I didn't figure out is exactly how the two gmem instances are bound to the same memslot.

The case I can imagine is

1. create gmem1 with GUEST_MEMFD_FLAG_MMAP and GUEST_MEMFD_FLAG_INIT_SHARED, and get fd1. mmap the returned fd1 to get a hva.

2. create gmem2 to get a fd2.

3. call KVM_SET_USER_MEMORY_REGION2 with KVM_MEM_GUEST_MEMFD flag. Pass the @hva from 1) to 'userspace_addr' field and pass the gmem fd2 to 'guest_memfd' field.

Quoted 5 lines
Quoted 5 lines

However, with this case, only gmem2 is bound to the memslot while gmem1 is not. Following PUNCH_HOLE on gmem1 doesn't invalidate any mappings because the f->bindings of it is empty.

Do I miss anything?

Oh, I misread your question. You were specifically asking about a PUNCH_HOLE in a SHARED gmem instance over-invalidating PRIVATE mappings. You're right, that wouldn't happen because the invalidations would come in via mmu_notifiers, not from KVM.

Quoted 1 line

I think Xiaoyao is right about this, the "vice versa" part is wrong,

Ya.

Quoted 3 lines

Is this just a case of user error? That with gmem_in_place_conversion, userspace should not use userspace_addr from something other than the gmem for the same memslot?

Yes. It's not just invalidations that will go sideways, KVM accesses to guest memory won't hit the same physical page as actual guest accesses.

Huh. But that isn't strictly guaranteed, because userspace could bind to a memslot that isn't configured with GUEST_MEMFD_FLAG_MMAP, in which case SHARED faults will go through the VMA, not kvm_mmu_faultin_pfn_gmem(). It's a bit early in the morning, but off the top of my head, I can't think of any reason we need to support such a setup. If userspace really, really wants to use a separate mapping, they could DELETE+CREATE an equivalent memslot without the guest_memfd file descriptor.

So I think we should do this?

virt/kvm/guest_memfd.c +1 -1
index 9c2d52bdf25e..86f53e53a136 100644
@@ -1013,7 +1013,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 */
 	WRITE_ONCE(slot->gmem.file, file);
 	slot->gmem.pgoff = start;
-	if (kvm_gmem_supports_mmap(inode))
+	if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
 		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
 
 	xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);

Regardless, the key aspect of all this is that in-place conversion is brand new functionality, so we don't have to ensure backwards compatibility.

#47 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

On 8/26/2026 9:53 PM, Sean Christopherson wrote:
<snip>

Quoted 8 lines
Quoted 3 lines

Is this just a case of user error? That with gmem_in_place_conversion, userspace should not use userspace_addr from something other than the gmem for the same memslot?

Yes. It's not just invalidations that will go sideways,

KVM accesses to guest
memory won't hit the same physical page as actual guest accesses.

Side topic.

If KVM enforces KVM_MEMSLOT_GMEM_ONLY when gmem_in_place_conversion is true like below proposal, should we update KVM's guest memory accessors to access gmem directly? If still use the existing code of accessing userspace_addr of the memslot, I think KVM needs to document clearly that when KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES is enumerated, if configuring the guest_memfd for a memslot, the userspace_addr passed in needs to be the mmaped address of the guest_memfd.

Quoted 15 lines

Huh. But that isn't strictly guaranteed, because userspace could bind to a memslot that isn't configured with GUEST_MEMFD_FLAG_MMAP, in which case SHARED faults will go through the VMA, not kvm_mmu_faultin_pfn_gmem(). It's a bit early in the morning, but off the top of my head, I can't think of any reason we need to support such a setup. If userspace really, really wants to use a separate mapping, they could DELETE+CREATE an equivalent memslot without the guest_memfd file descriptor.

So I think we should do this?

virt/kvm/guest_memfd.c +0 -0
index 9c2d52bdf25e..86f53e53a136 100644

virt/kvm/guest_memfd.c int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · line 1017

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -1
@@ -1013,7 +1013,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
  	 */
  	WRITE_ONCE(slot->gmem.file, file);
  	slot->gmem.pgoff = start;
-	if (kvm_gmem_supports_mmap(inode))
+	if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
  		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;

I like this idea. It makes gmem_in_place_conversion a step closer to what its name implies, though in-place conversion is not truly 100% guaranteed[*].

[*] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines
virt/kvm/guest_memfd.c +0 -0
  	xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);
 

Regardless, the key aspect of all this is that in-place conversion is brand new functionality, so we don't have to ensure backwards compatibility.

#48 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

Quoted 8 lines · Xiaoyao Li

On 8/26/2026 9:53 PM, Sean Christopherson wrote:
<snip>

Quoted 5 lines
Quoted 3 lines

Is this just a case of user error? That with gmem_in_place_conversion, userspace should not use userspace_addr from something other than the gmem for the same memslot?

Yes. It's not just invalidations that will go sideways,

Quoted 8 lines
Quoted 2 lines

KVM accesses to guest
memory won't hit the same physical page as actual guest accesses.

Side topic.

If KVM enforces KVM_MEMSLOT_GMEM_ONLY when gmem_in_place_conversion is true like below proposal, should we update KVM's guest memory accessors to access gmem directly?

No, because routing KVM accesses through uaccess means we just have to ensure PRIVATE memory isn't mapped into userspace, which we already have to do anyways.

Now, that doesn't preclude future work from optimizing certain flows to avoid GUP, or to harden KVM in other ways. But as a first step, bypassing uaccess is actively dangerous.

Quoted 5 lines

If still use the existing code of accessing userspace_addr of the memslot, I think KVM needs to document clearly that when KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES is enumerated, if configuring the guest_memfd for a memslot, the userspace_addr passed in needs to be the mmaped address of the guest_memfd.

Ya.

Quoted 27 lines
Quoted 15 lines

Huh. But that isn't strictly guaranteed, because userspace could bind to a memslot that isn't configured with GUEST_MEMFD_FLAG_MMAP, in which case SHARED faults will go through the VMA, not kvm_mmu_faultin_pfn_gmem(). It's a bit early in the morning, but off the top of my head, I can't think of any reason we need to support such a setup. If userspace really, really wants to use a separate mapping, they could DELETE+CREATE an equivalent memslot without the guest_memfd file descriptor.

So I think we should do this?

virt/kvm/guest_memfd.c +0 -0
index 9c2d52bdf25e..86f53e53a136 100644

virt/kvm/guest_memfd.c int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · line 1017

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -1
@@ -1013,7 +1013,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
  	 */
  	WRITE_ONCE(slot->gmem.file, file);
  	slot->gmem.pgoff = start;
-	if (kvm_gmem_supports_mmap(inode))
+	if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
  		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;

I like this idea. It makes gmem_in_place_conversion a step closer to what its name implies, though in-place conversion is not truly 100% guaranteed[*].

[*] https://lore.kernel.org/all/[email protected]

Reviewed-by on v10 9/41
#49 lore raw

Re: [PATCH v10 09/41] KVM: guest_memfd: Filter both shared and private when invalidating

In reply to Ackerley Tng
Quoted 12 lines · On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay…

From: Ackerley Tng <[email protected]>

Before conversion, a guest_memfd could be either all shared, or all private, configured at creation time using the INIT_SHARED flag. Hence, when zapping pages from stage 2 page tables, guest_memfd can filter which to zap based on the initial shared/private state.

With conversion, guest_memfd tracks shared/private state on a per-page level, so a range can contain both private and shared pages. Zap both private and shared pages for simplicity.

Quoted 6 lines

An alternative would be to iterate guest_memfd attributes and only zap both if shared and private pages exist within the range. Setting both the shared and private filters lets the zapping logic do that iteration instead.

Cheers,
/fuad

Quoted 76 lines
---
 virt/kvm/guest_memfd.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)
virt/kvm/guest_memfd.c +8 -17
index 75979c885e030..b1537226e5adf 100644
@@ -136,14 +136,6 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
        return folio;
 }
 
-static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
-{
-       if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
-               return KVM_FILTER_SHARED;
-
-       return KVM_FILTER_PRIVATE;
-}
-
 static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
                                        pgoff_t end,
                                        enum kvm_gfn_range_filter attr_filter)
@@ -186,15 +178,13 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 }
 
 static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
-                                     pgoff_t end)
+                                     pgoff_t end,
+                                     enum kvm_gfn_range_filter filter)
 {
-       enum kvm_gfn_range_filter attr_filter;
        struct gmem_file *f;
 
-       attr_filter = kvm_gmem_get_invalidate_filter(inode);
-
        kvm_gmem_for_each_file(f, inode)
-               __kvm_gmem_invalidate_start(f, start, end, attr_filter);
+               __kvm_gmem_invalidate_start(f, start, end, filter);
 }
 
 static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
@@ -229,7 +219,8 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
         */
        filemap_invalidate_lock(inode->i_mapping);
 
-       kvm_gmem_invalidate_start(inode, start, end);
+       kvm_gmem_invalidate_start(inode, start, end,
+                                 KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
 
        truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
 
@@ -344,7 +335,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
         * memory, as its lifetime is associated with the inode, not the file.
         */
        __kvm_gmem_invalidate_start(f, 0, -1ul,
-                                   kvm_gmem_get_invalidate_filter(inode));
+                                   KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
        __kvm_gmem_invalidate_end(f, 0, -1ul);
 
        list_del(&f->entry);
@@ -525,7 +516,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
        start = folio->index;
        end = start + folio_nr_pages(folio);
 
-       kvm_gmem_invalidate_start(mapping->host, start, end);
+       kvm_gmem_invalidate_start(mapping->host, start, end,
+                                 KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
 
        /*
         * Do not truncate the range, what action is taken in response to the
 
--

2.55.0.654.g21b8a5bc05-goog

#50 lore raw

PATCH v10 10/41 KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2

In reply to Ackerley Tng

Add a new ioctl (and matching struct), KVM_SET_MEMORY_ATTRIBUTES2, using the same base ioctl number (0xd2), but with R/W semantics for the kernel instead of just read semantics. "Officially" documenting that KVM writes to the payload will allow KVM to support partial/incremental conversions, instead of all-or-nothing updates (which requires complex unwinding), by recording the failing offset if an error occurs.

Opportunistically add a new struct as well, even though KVM could squeeze the error offset into "struct kvm_memory_attributes", as there's no cost to doing so in practice. Pad the struct with a pile of extra space to try and avoid ending up with "struct kvm_memory_attributes3" in the future. Use the same layout for the fields that common to version 1 of the struct, e.g. to ease upgrading userspace, and to provide flexibility if KVM ever adds support for KVM_SET_MEMORY_ATTRIBUTES2 at VM scope.

Introduce KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES to advertise the availability of the KVM_SET_MEMORY_ATTRIBUTES2 ioctl.

Update the KVM API documentation to define the new ioctl and its behavior, and add the necessary UAPI definitions and capability checks.

The process of setting memory attributes has a clear point of no return because, for CoCo VMs, zapping stage 2 page tables is a destructive operation. Unlike regular VMs, where re-faulting pages into the stage 2 page tables merely incurs a performance penalty, CoCo guests must (re-):accept pages after every fault. To preserve CoCo security guarantees, guests will not accept pages they did not explicitly request faults for. Consequently, during memory conversions, any operation that could cause the process to abort must be completed before the stage 2 page tables are zapped.

Zap only the ranges that are not already in the requested state to avoid inadvertently destroying (CoCo) data. ARM CCA guests will try to mark the entire DRAM as private at boot. If there are no shared pages at all, the to-private conversion can be skipped, but the existence of a single shared page would require the conversion process to proceed, and if it proceeds, zapping both shared and private pages would destroy data and break the guest.

---
 Documentation/virt/kvm/api.rst |  63 +++++++++++++++++++++-
 include/uapi/linux/kvm.h       |  15 ++++++
 virt/kvm/guest_memfd.c         | 119 +++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c            |  23 +++++---
 4 files changed, 212 insertions(+), 8 deletions(-)
Documentation/virt/kvm/api.rst +62 -1
index 62afc031897ce..1a3f664dbb197 100644
@@ -117,7 +117,7 @@ description:
       x86 includes both i386 and x86_64.
 
   Type:
-      system, vm, or vcpu.
+      system, vm, vcpu or guest_memfd.
 
   Parameters:
       what parameters are accepted by the ioctl.
@@ -6382,6 +6382,8 @@ S390:
 Returns -EINVAL if the VM has the KVM_VM_S390_UCONTROL flag set.
 Returns -EINVAL if called on a protected VM.
 
+.. _KVM_SET_MEMORY_ATTRIBUTES:
+
 4.141 KVM_SET_MEMORY_ATTRIBUTES
 -------------------------------
 
@@ -6575,6 +6577,65 @@ KVM_S390_KEYOP_SSKE
   Sets the storage key for the guest address ``guest_addr`` to the key
   specified in ``key``, returning the previous value in ``key``.
 
+4.145 KVM_SET_MEMORY_ATTRIBUTES2
+---------------------------------
+
+:Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES
+:Architectures: all
+:Type: guest_memfd ioctl
+:Parameters: struct kvm_memory_attributes2 (in)
+:Returns: 0 on success, <0 on error
+
+Errors:
+
+  ========== ===============================================================
+  EINVAL     The specified `offset` or `size` was invalid (e.g. not
+             page aligned, causes an overflow, or size is zero).
+  EFAULT     The parameter address was invalid.
+  ENOMEM     Ran out of memory trying to track private/shared state
+  ========== ===============================================================
+
+KVM_SET_MEMORY_ATTRIBUTES2 is an extension to
+KVM_SET_MEMORY_ATTRIBUTES that supports returning (writing) values to
+userspace.  The original (pre-extension) fields are shared with
+KVM_SET_MEMORY_ATTRIBUTES identically.
+
+Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES.
+
+::
+
+  struct kvm_memory_attributes2 {
+	union {
+		__u64 address;
+		__u64 offset;
+	};
+	__u64 size;
+	__u64 attributes;
+	__u64 flags;
+	__u64 reserved[12];
+  };
+
+  #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
+
+Set attributes for a range of offsets within a guest_memfd to
+KVM_MEMORY_ATTRIBUTE_PRIVATE to limit the specified guest_memfd backed
+memory range for guest use. Even if KVM_CAP_GUEST_MEMFD_MMAP is
+supported, after a successful call to set
+KVM_MEMORY_ATTRIBUTE_PRIVATE, the requested range will not be mappable
+into host userspace and will only be mappable by the guest.
+
+To allow the range to be mappable into host userspace again, call
+KVM_SET_MEMORY_ATTRIBUTES2 on the guest_memfd again with
+KVM_MEMORY_ATTRIBUTE_PRIVATE unset.
+
+KVM does not directly manipulate the memory contents of pages during
+attribute updates. However, the process of setting these attributes,
+which includes operations such as unmapping pages from the host or
+stage-2 page tables, may result in side effects on memory contents
+that vary across different trusted firmware implementations.
+
+See also: :ref: `KVM_SET_MEMORY_ATTRIBUTES`.
+
 .. _kvm_run:
 
 5. The kvm_run structure
include/uapi/linux/kvm.h +15 -0
index 419011097fa8e..80985e28e3b21 100644
@@ -997,6 +997,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 250
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1649,6 +1650,20 @@ struct kvm_memory_attributes {
 	__u64 flags;
 };
 
+/* Available with KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES */
+#define KVM_SET_MEMORY_ATTRIBUTES2              _IOWR(KVMIO,  0xd2, struct kvm_memory_attributes2)
+
+struct kvm_memory_attributes2 {
+	union {
+		__u64 address;
+		__u64 offset;
+	};
+	__u64 size;
+	__u64 attributes;
+	__u64 flags;
+	__u64 reserved[12];
+};
+
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
virt/kvm/guest_memfd.c +119 -0

130 lines not on this page · raw message

virt/kvm/kvm_main.c +16 -7
index 0a5a1bee8de0f..c4c09593ef8b9 100644
@@ -2423,18 +2423,22 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
 
+#ifdef kvm_arch_has_private_mem
+static u64 kvm_supports_private_mem(struct kvm *kvm)
+{
+	return !kvm || kvm_arch_has_private_mem(kvm);
+}
+#else
+#define kvm_supports_private_mem(kvm) false
+#endif
+
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
-#ifdef kvm_arch_has_private_mem
-	if (gmem_in_place_conversion)
+	if (gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
 		return 0;
 
-	if (!kvm || kvm_arch_has_private_mem(kvm))
-		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
-#endif
-
-	return 0;
+	return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 
 /*
@@ -4976,6 +4980,11 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 		return 1;
 	case KVM_CAP_GUEST_MEMFD_FLAGS:
 		return kvm_gmem_get_supported_flags(kvm);
+	case KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES:
+		if (!gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
+			return 0;
+
+		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 #endif
 	default:
 		break;
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#51 lore raw

PATCH v10 11/41 KVM: guest_memfd: Ensure pages are not in use before conversion

In reply to Ackerley Tng

When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.

guest_memfd checks for unexpected refcounts to determine whether a page is still in use. The only expected refcounts after unmapping the range requested for conversion are those that are held by guest_memfd itself.

Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed to userspace. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.

Update documentation to document the error_offset field and the possible -EAGAIN error.

---
 Documentation/virt/kvm/api.rst | 19 ++++++++++--
 include/uapi/linux/kvm.h       |  3 +-
 virt/kvm/guest_memfd.c         | 66 ++++++++++++++++++++++++++++++++++++++----
 3 files changed, 80 insertions(+), 8 deletions(-)
Documentation/virt/kvm/api.rst +17 -2
index 1a3f664dbb197..1e64026d7c1e9 100644
@@ -6583,7 +6583,7 @@ KVM_S390_KEYOP_SSKE
 :Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES
 :Architectures: all
 :Type: guest_memfd ioctl
-:Parameters: struct kvm_memory_attributes2 (in)
+:Parameters: struct kvm_memory_attributes2 (in/out)
 :Returns: 0 on success, <0 on error
 
 Errors:
@@ -6592,6 +6592,8 @@ Errors:
   EINVAL     The specified `offset` or `size` was invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   EFAULT     The parameter address was invalid.
+  EAGAIN     Some page within requested range had unexpected refcounts. The
+             offset of the page will be returned in `error_offset`.
   ENOMEM     Ran out of memory trying to track private/shared state
   ========== ===============================================================
 
@@ -6605,6 +6607,7 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES.
 ::
 
   struct kvm_memory_attributes2 {
+	/* in */
 	union {
 		__u64 address;
 		__u64 offset;
@@ -6612,7 +6615,9 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES.
 	__u64 size;
 	__u64 attributes;
 	__u64 flags;
-	__u64 reserved[12];
+	/* out */
+	__u64 error_offset;
+	__u64 reserved[11];
   };
 
   #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
@@ -6634,6 +6639,16 @@ which includes operations such as unmapping pages from the host or
 stage-2 page tables, may result in side effects on memory contents
 that vary across different trusted firmware implementations.
 
+If this ioctl returns -EAGAIN, the offset of the page with unexpected
+refcounts will be returned in `error_offset`. This can occur if there
+are transient refcounts on the pages, taken by other parts of the
+kernel.
+
+Userspace is expected to figure out how to remove all known refcounts
+on the shared pages, such as refcounts taken by get_user_pages(), and
+try the ioctl again. A possible source of these long term refcounts is
+if the guest_memfd memory was pinned in IOMMU page tables.
+
 See also: :ref: `KVM_SET_MEMORY_ATTRIBUTES`.
 
 .. _kvm_run:
include/uapi/linux/kvm.h +2 -1
index 80985e28e3b21..129d6f6303251 100644
@@ -1661,7 +1661,8 @@ struct kvm_memory_attributes2 {
 	__u64 size;
 	__u64 attributes;
 	__u64 flags;
-	__u64 reserved[12];
+	__u64 error_offset;
+	__u64 reserved[11];
 };
 
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
virt/kvm/guest_memfd.c +61 -5

94 lines not on this page · raw message

Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#52 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 157 lines · On Fri, Aug 07, 2026 at 02:52:50PM -0700, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.

guest_memfd checks for unexpected refcounts to determine whether a page is still in use. The only expected refcounts after unmapping the range requested for conversion are those that are held by guest_memfd itself.

Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed to userspace. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.

Update documentation to document the error_offset field and the possible -EAGAIN error.

---
 Documentation/virt/kvm/api.rst | 19 ++++++++++--
 include/uapi/linux/kvm.h       |  3 +-
 virt/kvm/guest_memfd.c         | 66 ++++++++++++++++++++++++++++++++++++++----
 3 files changed, 80 insertions(+), 8 deletions(-)
Documentation/virt/kvm/api.rst +17 -2
index 1a3f664dbb197..1e64026d7c1e9 100644
@@ -6583,7 +6583,7 @@ KVM_S390_KEYOP_SSKE
 :Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES
 :Architectures: all
 :Type: guest_memfd ioctl
-:Parameters: struct kvm_memory_attributes2 (in)
+:Parameters: struct kvm_memory_attributes2 (in/out)
 :Returns: 0 on success, <0 on error
 
 Errors:
@@ -6592,6 +6592,8 @@ Errors:
   EINVAL     The specified `offset` or `size` was invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   EFAULT     The parameter address was invalid.
+  EAGAIN     Some page within requested range had unexpected refcounts. The
+             offset of the page will be returned in `error_offset`.
   ENOMEM     Ran out of memory trying to track private/shared state
   ========== ===============================================================
 
@@ -6605,6 +6607,7 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES.
 ::
 
   struct kvm_memory_attributes2 {
+	/* in */
 	union {
 		__u64 address;
 		__u64 offset;
@@ -6612,7 +6615,9 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES.
 	__u64 size;
 	__u64 attributes;
 	__u64 flags;
-	__u64 reserved[12];
+	/* out */
+	__u64 error_offset;
+	__u64 reserved[11];
   };
 
   #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
@@ -6634,6 +6639,16 @@ which includes operations such as unmapping pages from the host or
 stage-2 page tables, may result in side effects on memory contents
 that vary across different trusted firmware implementations.
 
+If this ioctl returns -EAGAIN, the offset of the page with unexpected
+refcounts will be returned in `error_offset`. This can occur if there
+are transient refcounts on the pages, taken by other parts of the
+kernel.
+
+Userspace is expected to figure out how to remove all known refcounts
+on the shared pages, such as refcounts taken by get_user_pages(), and
+try the ioctl again. A possible source of these long term refcounts is
+if the guest_memfd memory was pinned in IOMMU page tables.
+
 See also: :ref: `KVM_SET_MEMORY_ATTRIBUTES`.
 
 .. _kvm_run:
include/uapi/linux/kvm.h +2 -1
index 80985e28e3b21..129d6f6303251 100644
@@ -1661,7 +1661,8 @@ struct kvm_memory_attributes2 {
 	__u64 size;
 	__u64 attributes;
 	__u64 flags;
-	__u64 reserved[12];
+	__u64 error_offset;
+	__u64 reserved[11];
 };
 
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
virt/kvm/guest_memfd.c +35 -2
index 3783e63476569..13c3989136f67 100644
@@ -524,8 +524,42 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
 	return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
+static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
+					    size_t nr_pages, pgoff_t *err_index)
+{
+	struct address_space *mapping = inode->i_mapping;
+	const int filemap_get_folios_refcount = 1;
+	pgoff_t last = start + nr_pages - 1;
+	struct folio_batch fbatch;
+	bool safe = true;
+	pgoff_t next;
+	int i;
+
+	folio_batch_init(&fbatch);
+
+	next = start;
+	while (safe && filemap_get_folios(mapping, &next, last, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+
+			if (folio_ref_count(folio) !=
+			    folio_nr_pages(folio) + filemap_get_folios_refcount) {
+				safe = false;
+				*err_index = max(start, folio->index);
+				break;
+			}
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+
+	return safe;
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs)
+				     size_t nr_pages, uint64_t attrs,
+				     pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
virt/kvm/guest_memfd.c +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Quoted 44 lines
virt/kvm/guest_memfd.c +17 -3
+			mas_destroy(&mas);
+			r = -EAGAIN;
+			goto out;
+		}
+	}
 
 	/*
 	 * From this point on guest_memfd has performed necessary
@@ -564,9 +611,10 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp)
 	struct gmem_file *f = file->private_data;
 	struct inode *inode = file_inode(file);
 	struct kvm_memory_attributes2 attrs;
+	pgoff_t err_index;
 	size_t nr_pages;
 	pgoff_t index;
-	int i;
+	int i, r;
 
 	if (copy_from_user(&attrs, argp, sizeof(attrs)))
 		return -EFAULT;
@@ -592,8 +640,16 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp)
 
 	nr_pages = attrs.size >> PAGE_SHIFT;
 	index = attrs.offset >> PAGE_SHIFT;
-	return __kvm_gmem_set_attributes(inode, index, nr_pages,
-					 attrs.attributes);
+	r = __kvm_gmem_set_attributes(inode, index, nr_pages, attrs.attributes,
+				      &err_index);
+	if (r) {
+		attrs.error_offset = ((uint64_t)err_index) << PAGE_SHIFT;
+
+		if (copy_to_user(argp, &attrs, sizeof(attrs)))
+			return -EFAULT;
+	}
+
+	return r;
 }
 
 static long kvm_gmem_ioctl(struct file *file, unsigned int ioctl,
Signature · 2 lines
-- 
2.55.0.654.g21b8a5bc05-goog
#53 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 77 lines · Yan Zhao
Quoted 45 lines · On Fri, Aug 07, 2026 at 02:52:50PM -0700, Ackerley Tng via B4 Relay wrote:
(quoted diff) +35 -2
+static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
+					    size_t nr_pages, pgoff_t *err_index)
+{
+	struct address_space *mapping = inode->i_mapping;
+	const int filemap_get_folios_refcount = 1;
+	pgoff_t last = start + nr_pages - 1;
+	struct folio_batch fbatch;
+	bool safe = true;
+	pgoff_t next;
+	int i;
+
+	folio_batch_init(&fbatch);
+
+	next = start;
+	while (safe && filemap_get_folios(mapping, &next, last, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+
+			if (folio_ref_count(folio) !=
+			    folio_nr_pages(folio) + filemap_get_folios_refcount) {
+				safe = false;
+				*err_index = max(start, folio->index);
+				break;
+			}
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+
+	return safe;
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs)
+				     size_t nr_pages, uint64_t attrs,
+				     pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);
Quoted 7 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
Signature · 1 line
2.43.2
#54 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 33 lines · Yan Zhao

[...snip...]

Quoted 30 lines
Quoted 5 lines
(quoted diff) +0 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

Quoted 6 lines
Quoted 6 lines
                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Thanks for reporting this!

Quoted 4 lines
Quoted 4 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

IIUC CPU 1 isn't blocked by a conversion so stages A and B should complete fine, and CPU 0 would already be retrying for other reasons anyway, like speculative refcounts from elsewhere in the kernel, so the conversion would take longer but it'd work out.

Is that understanding right, that this doesn't completely break conversions?

Quoted 3 lines
Quoted 3 lines

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Not really sure about this, how will control go back to guest_memfd after the fault finishes for guest_memfd to unlock the filemap?

Quoted 46 lines

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -4
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
--
Quoted 1 line

2.43.2

Hmm going with the above CPU 0 and 1 illustration, if instead CPU 0 truncates the folio and the folio ends up being freed, then KVM's MMU has a pointer to a page that is already
freed. kvm_release_faultin_page() is passed the pointer to this page and will dereference the page.

#55 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 49 lines · Ackerley Tng
Quoted 33 lines · Yan Zhao

[...snip...]

Quoted 30 lines
Quoted 5 lines
(quoted diff) +0 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

Quoted 6 lines
Quoted 6 lines
                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Thanks for reporting this!

Quoted 4 lines
Quoted 4 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

Quoted 7 lines

IIUC CPU 1 isn't blocked by a conversion so stages A and B should complete fine, and CPU 0 would already be retrying for other reasons anyway, like speculative refcounts from elsewhere in the kernel, so the conversion would take longer but it'd work out.

Is that understanding right, that this doesn't completely break conversions?

It depends on the timing. The max retry count cannot be expected under unlucky conditions.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Not really sure about this, how will control go back to guest_memfd after the fault finishes for guest_memfd to unlock the filemap?

I don't understand your question. But I find this solution is less ideal than my below proposal.

Quoted 42 lines
Quoted 42 lines

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
Quoted 10 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0 -1
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
--

2.43.2

Hmm going with the above CPU 0 and 1 illustration, if instead CPU 0 truncates the folio and the folio ends up being freed, then KVM's MMU has a pointer to a page that is already
freed. kvm_release_faultin_page() is passed the pointer to this page and will dereference the page.

Not really. It's just like KVM mapping non-refcounted pages. kvm_release_faultin_page() does not access the non-refcounted pages. The invalidate protocol also ensures no mapping of stale pfn.

As below, if CPU 0 truncates the folio, it needs to hold filemap invalidate lock, add KVM mmu invalidate range, hold mmu_lock, zap KVM mappings before the truncation.

CPU 0 CPU 1
----- --------

                                         Save fault->mmu_seq

                                     B1. filemap_invalidate_lock_shared
                                         __kvm_gmem_get_pfn
                                         folio_put
                                         filemap_invalidate_unlock_shared

				     B2. read_lock
                                         is_page_fault_stale

                                     B3. kvm_tdp_mmu_map
                                     B4. kvm_mmu_finish_page_fault   
                                         read_unlock

A1. filemap_invalidate_lock
kvm_gmem_invalidate_start

A2. write_lock
zap KVM MMU
write_unlock
truncate

A3. kvm_gmem_invalidate_end
filemap_invalidate_unlock

A1 occurs either before or after B1.
1) If A1 occurs before B1, B1 will find the correct pfn.
2) If A1 occurs after B1 and before B2,
a. if A2 is before B2, B2 must find the fault is stale, so it's fine. b. if A2 is after B2, A2 must be after B4 as well. So, accessing stale pfn
in CPU 1 is fine.
3) If A1 occurs after B2 and before B3,
4) If A1 occurs after B3 and before B4,
5) If A1 occurs after B4,
A2 must be after B4 (for 3-5 conditions).
So, accessing stale pfn in CPU 1 is fine.

#56 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 60 lines · Yan Zhao
Quoted 49 lines · Ackerley Tng
Quoted 33 lines · Yan Zhao

[...snip...]

Quoted 30 lines
Quoted 5 lines
(quoted diff) +0 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

Quoted 6 lines
Quoted 6 lines
                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Thanks for reporting this!

Quoted 4 lines
Quoted 4 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

Quoted 7 lines

IIUC CPU 1 isn't blocked by a conversion so stages A and B should complete fine, and CPU 0 would already be retrying for other reasons anyway, like speculative refcounts from elsewhere in the kernel, so the conversion would take longer but it'd work out.

Is that understanding right, that this doesn't completely break conversions?

It depends on the timing. The max retry count cannot be expected under unlucky conditions.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Not really sure about this, how will control go back to guest_memfd after the fault finishes for guest_memfd to unlock the filemap?

I don't understand your question. But I find this solution is less ideal than my below proposal.

When kvm_gmem_get_pfn() is called from kvm_mmu_faultin_pfn_gmem(), KVM MMU takes over from there, there isn't another call when KVM MMU finishes mapping the page into the stage 2 page tables back into guest_memfd.

After stage B, how is filemap_invalidate_unlock_shared() going to be called? Do you mean filemap_invalidate_unlock_shared(folio->mapping)?

I think filemap_invalidate_unlock_shared(folio->mapping) is a little asymmetric...

Quoted 88 lines
Quoted 42 lines
Quoted 42 lines

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
Quoted 10 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0 -1
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
--

2.43.2

Hmm going with the above CPU 0 and 1 illustration, if instead CPU 0 truncates the folio and the folio ends up being freed, then KVM's MMU has a pointer to a page that is already
freed. kvm_release_faultin_page() is passed the pointer to this page and will dereference the page.

Not really. It's just like KVM mapping non-refcounted pages. kvm_release_faultin_page() does not access the non-refcounted pages. The invalidate protocol also ensures no mapping of stale pfn.

As below, if CPU 0 truncates the folio, it needs to hold filemap invalidate lock, add KVM mmu invalidate range, hold mmu_lock, zap KVM mappings before the truncation.

CPU 0 CPU 1
----- --------

                                         Save fault->mmu_seq

                                     B1. filemap_invalidate_lock_shared
                                         __kvm_gmem_get_pfn
                                         folio_put
                                         filemap_invalidate_unlock_shared

				     B2. read_lock
                                         is_page_fault_stale

                                     B3. kvm_tdp_mmu_map
                                     B4. kvm_mmu_finish_page_fault
                                         read_unlock

A1. filemap_invalidate_lock
kvm_gmem_invalidate_start

A2. write_lock
zap KVM MMU
write_unlock
truncate

A3. kvm_gmem_invalidate_end
filemap_invalidate_unlock

Quoted 11 lines

A1 occurs either before or after B1.
1) If A1 occurs before B1, B1 will find the correct pfn.
2) If A1 occurs after B1 and before B2,
a. if A2 is before B2, B2 must find the fault is stale, so it's fine. b. if A2 is after B2, A2 must be after B4 as well. So, accessing stale pfn
in CPU 1 is fine.
3) If A1 occurs after B2 and before B3,
4) If A1 occurs after B3 and before B4,
5) If A1 occurs after B4,
A2 must be after B4 (for 3-5 conditions).
So, accessing stale pfn in CPU 1 is fine.

It's not about the stale PFN, if there's no refcount on the page returned from B1, then after A3, the page can be freed.

Contractually, I think KVM is allowed to reference the page? kvm_release_page_clean() calls kvm_set_page_accessed() on the page (UAF?). Not sure which other KVM architectures reference the struct page itself. Are we going to teach the rest of KVM to not reference and not do kvm_release_page_*?

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

#57 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 72 lines · Ackerley Tng
Quoted 60 lines · Yan Zhao
Quoted 49 lines · Ackerley Tng
Quoted 33 lines · Yan Zhao

[...snip...]

Quoted 30 lines
Quoted 5 lines
(quoted diff) +0 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

Quoted 6 lines
Quoted 6 lines
                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Thanks for reporting this!

Quoted 4 lines
Quoted 4 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

Quoted 7 lines

IIUC CPU 1 isn't blocked by a conversion so stages A and B should complete fine, and CPU 0 would already be retrying for other reasons anyway, like speculative refcounts from elsewhere in the kernel, so the conversion would take longer but it'd work out.

Is that understanding right, that this doesn't completely break conversions?

It depends on the timing. The max retry count cannot be expected under unlucky conditions.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Not really sure about this, how will control go back to guest_memfd after the fault finishes for guest_memfd to unlock the filemap?

I don't understand your question. But I find this solution is less ideal than my below proposal.

Quoted 7 lines

When kvm_gmem_get_pfn() is called from kvm_mmu_faultin_pfn_gmem(), KVM MMU takes over from there, there isn't another call when KVM MMU finishes mapping the page into the stage 2 page tables back into guest_memfd.

After stage B, how is filemap_invalidate_unlock_shared() going to be called? Do you mean filemap_invalidate_unlock_shared(folio->mapping)?

Something like this. However, since this would cause the shared filemap invalidate lock to be held longer, conversions may have to wait for any on-going faults regardless of the GFN range, which I don't quite like.

Quoted 103 lines

I think filemap_invalidate_unlock_shared(folio->mapping) is a little asymmetric...

Quoted 88 lines
Quoted 42 lines
Quoted 42 lines

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
Quoted 10 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0 -1
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
--

2.43.2

Hmm going with the above CPU 0 and 1 illustration, if instead CPU 0 truncates the folio and the folio ends up being freed, then KVM's MMU has a pointer to a page that is already
freed. kvm_release_faultin_page() is passed the pointer to this page and will dereference the page.

Not really. It's just like KVM mapping non-refcounted pages. kvm_release_faultin_page() does not access the non-refcounted pages. The invalidate protocol also ensures no mapping of stale pfn.

As below, if CPU 0 truncates the folio, it needs to hold filemap invalidate lock, add KVM mmu invalidate range, hold mmu_lock, zap KVM mappings before the truncation.

CPU 0 CPU 1
----- --------

                                         Save fault->mmu_seq

                                     B1. filemap_invalidate_lock_shared
                                         __kvm_gmem_get_pfn
                                         folio_put
                                         filemap_invalidate_unlock_shared

				     B2. read_lock
                                         is_page_fault_stale

                                     B3. kvm_tdp_mmu_map
                                     B4. kvm_mmu_finish_page_fault
                                         read_unlock

A1. filemap_invalidate_lock
kvm_gmem_invalidate_start

A2. write_lock
zap KVM MMU
write_unlock
truncate

A3. kvm_gmem_invalidate_end
filemap_invalidate_unlock

Quoted 11 lines

A1 occurs either before or after B1.
1) If A1 occurs before B1, B1 will find the correct pfn.
2) If A1 occurs after B1 and before B2,
a. if A2 is before B2, B2 must find the fault is stale, so it's fine. b. if A2 is after B2, A2 must be after B4 as well. So, accessing stale pfn
in CPU 1 is fine.
3) If A1 occurs after B2 and before B3,
4) If A1 occurs after B3 and before B4,
5) If A1 occurs after B4,
A2 must be after B4 (for 3-5 conditions).
So, accessing stale pfn in CPU 1 is fine.

Quoted 5 lines

It's not about the stale PFN, if there's no refcount on the page returned from B1, then after A3, the page can be freed.

Contractually, I think KVM is allowed to reference the page? kvm_release_page_clean() calls kvm_set_page_accessed() on the page

In my patch, "*page = NULL;" is returned in kvm_gmem_get_pfn(). So, fault->refcounted_page is NULL. With it, kvm_release_faultin_page() does not access the faultin PFN or the page.
So, no worry about UAF.

Quoted 3 lines

(UAF?). Not sure which other KVM architectures reference the struct page itself. Are we going to teach the rest of KVM to not reference and not do kvm_release_page_*?

It's just like how KVM maps non-refcounted pages.
And guest_memfd actually asks consumers (like TDX) not to take page refcount. __kvm_gmem_populate() also puts the folio refcount before invoking filemap_invalidate_unlock().

Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest. I have no strong opinion on whether it should be fixed after this series lands. But the fix I proposed is quite small :)

#58 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 190 lines · Yan Zhao
Quoted 72 lines · Ackerley Tng
Quoted 60 lines · Yan Zhao
Quoted 49 lines · Ackerley Tng
Quoted 33 lines · Yan Zhao

[...snip...]

Quoted 30 lines
Quoted 5 lines
(quoted diff) +0 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)

Review static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 588

Quoted 10 lines
(quoted diff) +9 -0
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {
+		unmap_mapping_pages(mapping, start, nr_pages, false);
+
+		if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
+						     err_index)) {

Note: conversion failures could occur if another vCPU is attempting to map a GFN within this range.

CPU 0 (setting attributes) CPU 1 (attempting to map)
-------------------------- --------------------

                                 A: mmu_invalidate_retry_gfn_unsafe
                                    filemap_invalidate_lock_shared
                                    __kvm_gmem_get_pfn ==> folio refcount++
                                    filemap_invalidate_unlock_shared

filemap_invalidate_lock
filemap_get_folios
check folio_ref_count(folio) ==> Not match !!
filemap_invalidate_unlock

Quoted 6 lines
Quoted 6 lines
                                 B: read_lock(&vcpu->kvm->mmu_lock);
                                    is_page_fault_stale
                                    kvm_mmu_finish_page_fault ==>folio recount--
				    read_unlock(&vcpu->kvm->mmu_lock);

Thanks for reporting this!

Quoted 4 lines
Quoted 4 lines

Retrying in kvm_gmem_is_safe_for_conversion() or moving the invocation of kvm_mmu_invalidate_start() + kvm_mmu_invalidate_range_add() to an earlier position does not help as long as CPU 1 stays at stage A.

Quoted 7 lines

IIUC CPU 1 isn't blocked by a conversion so stages A and B should complete fine, and CPU 0 would already be retrying for other reasons anyway, like speculative refcounts from elsewhere in the kernel, so the conversion would take longer but it'd work out.

Is that understanding right, that this doesn't completely break conversions?

It depends on the timing. The max retry count cannot be expected under unlucky conditions.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

So, should we avoid this failure?
e.g., by moving filemap_invalidate_unlock_shared() from stage A to after stage B?

Not really sure about this, how will control go back to guest_memfd after the fault finishes for guest_memfd to unlock the filemap?

I don't understand your question. But I find this solution is less ideal than my below proposal.

Quoted 7 lines

When kvm_gmem_get_pfn() is called from kvm_mmu_faultin_pfn_gmem(), KVM MMU takes over from there, there isn't another call when KVM MMU finishes mapping the page into the stage 2 page tables back into guest_memfd.

After stage B, how is filemap_invalidate_unlock_shared() going to be called? Do you mean filemap_invalidate_unlock_shared(folio->mapping)?

Something like this. However, since this would cause the shared filemap invalidate lock to be held longer, conversions may have to wait for any on-going faults regardless of the GFN range, which I don't quite like.

Quoted 106 lines

I think filemap_invalidate_unlock_shared(folio->mapping) is a little asymmetric...

Quoted 88 lines
Quoted 42 lines
Quoted 42 lines

Or what about having KVM always treat gmem page as non-refcounted, and have kvm_gmem_get_pfn() put folio refcount before releasing the filemap invalidate lock?
Below patch is applied and tested at the end of this series.

From 8c2f29bc15bceb6a8fa103cf2585ec11354fd74e Mon Sep 17 00:00:00 2001
From: Yan Zhao <[email protected]>
Date: Mon, 10 Aug 2026 06:24:52 +0800
Subject: [PATCH] KVM: guest_memfd: Return gmem page as non-refcounted

Have kvm_gmem_get_pfn() put gmem page refcount before releasing filemap invalidate lock and return the gmem page as non-refcounted. This avoids gmem memory attribute conversion failure caused by temporarily holding gmem page after faulting and before completing mapping.

guest_memfd always holds gmem page in filemap cache. TDX does not increment gmem page refcount when having gmem pages mapped in S-EPT. Additionally, as gmem pages are not swappable, setting dirty or accessed bit is not necessary. Therefore, there's no need to treat gmem pages as refcounted pages.

---
 virt/kvm/guest_memfd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
index 2115e73e455a..e357b4ffa777 100644
@@ -1332,11 +1332,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
+	folio_put(folio);
 
 	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
+		*page = NULL;
 
Quoted 10 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0 -1
 out:
 	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
--

2.43.2

Hmm going with the above CPU 0 and 1 illustration, if instead CPU 0 truncates the folio and the folio ends up being freed, then KVM's MMU has a pointer to a page that is already
freed. kvm_release_faultin_page() is passed the pointer to this page and will dereference the page.

Not really. It's just like KVM mapping non-refcounted pages. kvm_release_faultin_page() does not access the non-refcounted pages. The invalidate protocol also ensures no mapping of stale pfn.

As below, if CPU 0 truncates the folio, it needs to hold filemap invalidate lock, add KVM mmu invalidate range, hold mmu_lock, zap KVM mappings before the truncation.

CPU 0 CPU 1
----- --------

                                         Save fault->mmu_seq

                                     B1. filemap_invalidate_lock_shared
                                         __kvm_gmem_get_pfn
                                         folio_put
                                         filemap_invalidate_unlock_shared

				     B2. read_lock
                                         is_page_fault_stale

                                     B3. kvm_tdp_mmu_map
                                     B4. kvm_mmu_finish_page_fault
                                         read_unlock

A1. filemap_invalidate_lock
kvm_gmem_invalidate_start

A2. write_lock
zap KVM MMU
write_unlock
truncate

A3. kvm_gmem_invalidate_end
filemap_invalidate_unlock

Quoted 11 lines

A1 occurs either before or after B1.
1) If A1 occurs before B1, B1 will find the correct pfn.
2) If A1 occurs after B1 and before B2,
a. if A2 is before B2, B2 must find the fault is stale, so it's fine. b. if A2 is after B2, A2 must be after B4 as well. So, accessing stale pfn
in CPU 1 is fine.
3) If A1 occurs after B2 and before B3,
4) If A1 occurs after B3 and before B4,
5) If A1 occurs after B4,
A2 must be after B4 (for 3-5 conditions).
So, accessing stale pfn in CPU 1 is fine.

It's not about the stale PFN, if there's no refcount on the page returned from B1, then after A3, the page can be freed.

Quoted 7 lines
Quoted 2 lines

Contractually, I think KVM is allowed to reference the page? kvm_release_page_clean() calls kvm_set_page_accessed() on the page

In my patch, "*page = NULL;" is returned in kvm_gmem_get_pfn(). So, fault->refcounted_page is NULL. With it, kvm_release_faultin_page() does not access the faultin PFN or the page.
So, no worry about UAF.

Oh sorry, I missed that. I did see *page = NULL but I didn't connect the dots.

Wait but in the above trace, if CPU 1 got the page correctly, wouldnt it take the !r branch where page is not set to NULL? It's only later that CPU 0 truncates the page, but by that time KVM MMU already holds a non-NULL page pointer?

Unless on both branches guest_memfd just doesn't provide the page to KVM MMU?

Quoted 8 lines
Quoted 3 lines

(UAF?). Not sure which other KVM architectures reference the struct page itself. Are we going to teach the rest of KVM to not reference and not do kvm_release_page_*?

It's just like how KVM maps non-refcounted pages.
And guest_memfd actually asks consumers (like TDX) not to take page refcount. __kvm_gmem_populate() also puts the folio refcount before invoking filemap_invalidate_unlock().

I guess long term this is what guest_memfd would like, that KVM should just be getting PFNs from guest_memfd.

Quoted 4 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Which stress selftest is this? I can try running this on my side too.

Quoted 2 lines

I have no strong opinion on whether it should be fixed after this series lands. But the fix I proposed is quite small :)

Small code change, big implications! Might be nice to have this for the future where guest_memfd no longer hands out pages.

#59 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 7 lines · Ackerley Tng
Quoted 5 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Which stress selftest is this? I can try running this on my side too.

We have some selftests that are built on the basic TDX selftests. One just hammers the MMU stuff with a bunch of zaps and also weird stuff from the guest. It was eventually too much work to try to keep the internal enhancements rebased nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

#60 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 7 lines · Rick P
Quoted 6 lines · Ackerley Tng
Quoted 5 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Quoted 5 lines
Quoted 1 line

Which stress selftest is this? I can try running this on my side too.

We have some selftests that are built on the basic TDX selftests. One just hammers the MMU stuff with a bunch of zaps and also weird stuff from the guest. It was eventually too much work to try to keep the internal enhancements rebased

Would like all the comments we can get on TDX selftests v14 [1]!

Quoted 5 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

[1] https://lore.kernel.org/all/[email protected]/

#61 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 9 lines · Ackerley Tng
Quoted 7 lines · Rick P
Quoted 6 lines · Ackerley Tng
Quoted 5 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Which stress selftest is this? I can try running this on my side too.

We have some selftests that are built on the basic TDX selftests. One just hammers the MMU stuff with a bunch of zaps and also weird stuff from the guest. It was eventually too much work to try to keep the internal enhancements rebased

Would like all the comments we can get on TDX selftests v14 [1]!

I think we had a few. Let me try to round up some more folks.

Quoted 7 lines
Quoted 6 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

Quoted 6 lines

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

I'd point to Yan for what side of that line this is on, as she has been following this stuff much more to me.

#62 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 31 lines · Rick P Edgecombe
Quoted 9 lines · Ackerley Tng
Quoted 7 lines · Rick P
Quoted 6 lines · Ackerley Tng
Quoted 5 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Which stress selftest is this? I can try running this on my side too.

We have some selftests that are built on the basic TDX selftests. One just hammers the MMU stuff with a bunch of zaps and also weird stuff from the guest. It was eventually too much work to try to keep the internal enhancements rebased

Would like all the comments we can get on TDX selftests v14 [1]!

I think we had a few. Let me try to round up some more folks.

Quoted 11 lines
Quoted 6 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

Quoted 5 lines
Quoted 2 lines

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

This isn't uABI. You're talking about hitting a race condition between one task converting a page and another faulting in the same page. An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

That said, I am 100% in favor of not handing the caller a struct page. Now that the TDX APIs no longer require one, it's more than feasible. But, we absolutely shouldn't just nullify the pointer, we should drop the param entirely. Not just because it's cleaner, but because it also forces an audit of the callers to see if they subtly require a refcount (spoiler alert).

The lone holdout at this point is sev_handle_rmp_fault(), which could end up PSMASH-ing a PFN that has since been freed by KVM. Assuming holding mmu_lock while doing RMP operations is ok, something like the below? Completely untested.

As for in-place conversion, this is not a blocker.

arch/arm64/kvm/mmu.c +1 -1
index 6c941aaa10c6..8ef16ccf26ce 100644
@@ -1641,7 +1641,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 	/* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */
 	smp_rmb();
 
-	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, gfn, &pfn, &page, NULL);
+	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, gfn, &pfn, NULL);
 	if (ret) {
 		kvm_prepare_memory_fault_exit(s2fd->vcpu, s2fd->fault_ipa, PAGE_SIZE,
 					      write_fault, exec_fault, false);
arch/arm64/kvm/nested.c +1 -1
index fb54f6dad995..c982a6454fc9 100644
@@ -1411,7 +1411,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
 		if (is_error_noslot_pfn(pfn) || (write_fault && !writable))
 			return -EFAULT;
 	} else {
-		ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, &page, NULL);
+		ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, NULL);
 		if (ret) {
 			kvm_prepare_memory_fault_exit(vcpu, vt->wr.pa, PAGE_SIZE,
 					      write_fault, false, false);
arch/x86/kvm/mmu/mmu.c +1 -2
index c519e8e8d646..f0da212ab108 100644
@@ -4603,8 +4603,7 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 		return -EFAULT;
 	}
 
-	r = kvm_gmem_get_pfn(vcpu->kvm, fault->slot, fault->gfn, &fault->pfn,
-			     &fault->refcounted_page, &max_order);
+	r = kvm_gmem_get_pfn(vcpu->kvm, fault->slot, fault->gfn, &fault->pfn, &max_order);
 	if (r) {
 		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
 		return r;
arch/x86/kvm/svm/sev.c +23 -17
index fcb41dfde4c0..3b1c42e03deb 100644
@@ -4060,7 +4060,7 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	 * The new VMSA will be private memory guest memory, so retrieve the
 	 * PFN from the gmem backend.
 	 */
-	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL))
+	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, NULL))
 		return;
 
 	read_lock(&kvm->mmu_lock);
@@ -5003,7 +5003,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = vcpu->kvm;
 	int order, rmp_level, ret;
-	struct page *page;
+	unsigned long mmu_seq;
 	bool assigned;
 	kvm_pfn_t pfn;
 	gfn_t gfn;
@@ -5030,7 +5030,10 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 		return;
 	}
 
-	ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &page, &order);
+	mmu_seq = kvm->mmu_invalidate_seq;
+	smp_rmb();
+
+	ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &order);
 	if (ret) {
 		pr_warn_ratelimited("SEV: Unexpected RMP fault, no backing page for private GPA 0x%llx\n",
 				    gpa);
@@ -5041,7 +5044,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 	if (ret || !assigned) {
 		pr_warn_ratelimited("SEV: Unexpected RMP fault, no assigned RMP entry found for GPA 0x%llx PFN 0x%llx error %d\n",
 				    gpa, pfn, ret);
-		goto out_no_trace;
+		return;
 	}
 
 	/*
@@ -5069,26 +5072,29 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 	if (rmp_level == PG_LEVEL_4K)
 		goto out;
 
-	ret = snp_rmptable_psmash(pfn);
-	if (ret) {
-		/*
-		 * Look it up again. If it's 4K now then the PSMASH may have
-		 * raced with another process and the issue has already resolved
-		 * itself.
-		 */
-		if (!snp_lookup_rmpentry(pfn, &assigned, &rmp_level) &&
-		    assigned && rmp_level == PG_LEVEL_4K)
+	scoped_guard(read_lock)(&kvm->mmu_lock) {
+		if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
 			goto out;
 
-		pr_warn_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n",
-				    gpa, pfn, ret);
+		ret = snp_rmptable_psmash(pfn);
+		if (ret) {
+			/*
+			 * Look it up again. If it's 4K now then the PSMASH may have
+			 * raced with another process and the issue has already
+			 * resolved itself.
+			 */
+			if (!snp_lookup_rmpentry(pfn, &assigned, &rmp_level) &&
+		    	    assigned && rmp_level == PG_LEVEL_4K)
+				goto out;
+
+			pr_warn_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n",
+					    gpa, pfn, ret);
+		}
 	}
 
 	kvm_zap_gfn_range(kvm, gfn, gfn + PTRS_PER_PMD);
 out:
 	trace_kvm_rmp_fault(vcpu, gpa, pfn, error_code, rmp_level, ret);
-out_no_trace:
-	kvm_release_page_unused(page);
 }
 
 static bool is_pfn_range_shared(kvm_pfn_t start, kvm_pfn_t end)
include/linux/kvm_host.h +2 -4
index 03bfc92864b6..502465119ca0 100644
@@ -2586,13 +2586,11 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
-		     int *max_order);
+		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order);
 #else
 static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 				   struct kvm_memory_slot *slot, gfn_t gfn,
-				   kvm_pfn_t *pfn, struct page **page,
-				   int *max_order)
+				   kvm_pfn_t *pfn, int *max_order)
 {
 	KVM_BUG_ON(1, kvm);
 	return -EIO;
virt/kvm/guest_memfd.c +2 -8
index b596486d184c..ba7b46c1aa15 100644
@@ -751,8 +751,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 }
 
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
-		     int *max_order)
+		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
 {
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
 	struct folio *folio;
@@ -780,12 +779,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 #endif
 
 	folio_unlock(folio);
-
-	if (!r)
-		*page = folio_file_page(folio, index);
-	else
-		folio_put(folio);
-
+	folio_put(folio);
 	return r;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn);
#63 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 7 lines · Sean Christopherson
Quoted 2 lines

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

This isn't uABI.  You're talking about hitting a race condition between one task converting a page and another faulting in the same page.  An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

Sorry, I wasn't deeply following this thread. I was thinking along the lines of the scenario somehow affecting the uABI decisions around EAGAIN.

But just mostly wanted to chime in about the status of the tdx mmu stress test.

#64 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

In reply to Sean Christopherson
Quoted 32 lines · Sean Christopherson
Quoted 31 lines · Rick P Edgecombe
Quoted 9 lines · Ackerley Tng
Quoted 7 lines · Rick P
Quoted 6 lines · Ackerley Tng
Quoted 5 lines
Quoted 2 lines

Would like to see what Sean thinks of this. Either way, is it okay to follow up after conversions lands?

Let's see what Sean thinks of this :)
I raised this because the issue was encountered by one TDX's stress selftest.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Which stress selftest is this? I can try running this on my side too.

We have some selftests that are built on the basic TDX selftests. One just hammers the MMU stuff with a bunch of zaps and also weird stuff from the guest. It was eventually too much work to try to keep the internal enhancements rebased

Would like all the comments we can get on TDX selftests v14 [1]!

I think we had a few. Let me try to round up some more folks.

Quoted 11 lines
Quoted 6 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

This isn't uABI. You're talking about hitting a race condition between one task

Hmm. Perhaps it is not a uABI issue, since users are allowed to retry. However, it is hard to convince me that it makes sense to require users to retry a private-to-shared conversion before a GFN has ever been mapped, given that a retry is not required when the GFN is currently in use by the guest.

Quoted 3 lines

converting a page and another faulting in the same page. An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

Could you elaborate on how an NMI, SMI, or IRQ at just the right/wrong time could lead to the same test failures?

Do you mean they can cause a fault to be retried? Our test failure is an EAGAIN returned from a private-to-shared conversion before the page has even been mapped as private.

Quoted 5 lines

That said, I am 100% in favor of not handing the caller a struct page. Now that the TDX APIs no longer require one, it's more than feasible. But, we absolutely shouldn't just nullify the pointer, we should drop the param entirely. Not just because it's cleaner, but because it also forces an audit of the callers to see if they subtly require a refcount (spoiler alert).

Yeah, I also considered dropping the param entirely and was terrified by the lines of changes :)

If you are in favor of not handing the caller a struct page, the following changes should also be required on top of your change.

arch/arm64/kvm/mmu.c +0 -2
index 8ef16ccf26ce..d5aa197d2cbf 100644
@@ -1613,7 +1613,6 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
 	struct kvm_pgtable *pgt = s2fd->vcpu->arch.hw_mmu->pgt;
 	unsigned long mmu_seq;
-	struct page *page;
 	struct kvm *kvm = s2fd->vcpu->kvm;
 	void *memcache = NULL;
 	kvm_pfn_t pfn;
@@ -1681,7 +1680,6 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 	}
 
 out_unlock:
-	kvm_release_faultin_page(kvm, page, !!ret, prot & KVM_PGTABLE_PROT_W);
 	kvm_fault_unlock(kvm);
 
 	if ((prot & KVM_PGTABLE_PROT_W) && !ret)
arch/arm64/kvm/nested.c +1 -1
index c982a6454fc9..43523bb17621 100644
@@ -1360,7 +1360,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
 	bool write_fault, writable;
 	unsigned long mmu_seq;
 	struct vncr_tlb *vt;
-	struct page *page;
+	struct page *page = NULL;
 	u64 va, pfn, gfn;
 	int ret;
 
arch/x86/kvm/svm/sev.c +0 -3
index 3d3eb8387cd0..c5ba2c8cad74 100644
@@ -4017,7 +4017,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	struct kvm *kvm = vcpu->kvm;
 	gfn_t gfn = gpa_to_gfn(gpa);
 	unsigned long mmu_seq;
-	struct page *page;
 	kvm_pfn_t pfn;
 
 	lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex);
@@ -4077,8 +4076,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	else
 		svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
 	read_unlock(&kvm->mmu_lock);
-
-	kvm_release_page_clean(page);
 }
 
 /*
Quoted 3 lines

The lone holdout at this point is sev_handle_rmp_fault(), which could end up PSMASH-ing a PFN that has since been freed by KVM. Assuming holding mmu_lock while doing RMP operations is ok, something like the below? Completely untested.

Tested successfully after applying the above fix and the typo correction.

@@ -5073,7 +5070,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
if (rmp_level == PG_LEVEL_4K)
goto out;

- scoped_guard(read_lock)(&kvm->mmu_lock) {
+ scoped_guard(read_lock, &kvm->mmu_lock) {
if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
goto out;

Quoted 1 line

As for in-place conversion, this is not a blocker.

Sorry. I didn't intend to block in-place conversion. I encountered this issue during testing, so reported it.

#65 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 24 lines · Yan Zhao
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Rick P Edgecombe
Quoted 10 lines
Quoted 5 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

This isn't uABI. You're talking about hitting a race condition between one task

Hmm. Perhaps it is not a uABI issue, since users are allowed to retry. However, it is hard to convince me that it makes sense to require users to retry a private-to-shared conversion before a GFN has ever been mapped, given that a retry is not required when the GFN is currently in use by the guest.

Quoted 5 lines
Quoted 3 lines

converting a page and another faulting in the same page. An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

Could you elaborate on how an NMI, SMI, or IRQ at just the right/wrong time could lead to the same test failures?

Ah, sorry, my bad. I was speed reading and missed that the key to your suggested "*page = NULL" change was that the reference was put _before_ filemap_invalidate_unlock_shared(), i.e. before dropping
the invalidate lock and thus before __kvm_gmem_set_attributes() will walk the folios to look for outstanding references. I was thinking that putting the reference right away was just shrinking the timing window, but putting the reference while still holding the invalidate lock closes the window entirely.

So, I take back what I said about this not being ABI, and about this not blocking in-place conversion. It most definitely affects ABI, and so needs to be addressed before merging in-place conversion. The only question is if we want to commit to guaranteeing that conversion will succeed in this scenario, or if we want to take the easy way out and formally document that conversion can fail with EAGAIN at any time, even if userspace has never mmap()'d the memory in question.

I'm leaning pretty strongly towards guaranteeing conversion will succeed. We'll still need to document the EAGAIN behavior, but IMO there's a massive difference between conversion failing if there's a lingering reference acquired via a VMA, conversion failing because a vCPU page fault raced with conversion. E.g. being able to assert success in a very curated test, as the stress test presumably does, would be extremely valuable for helping detect/prevent edge case bugs.

The argument against guaranteeing success is that we might make our future lives harder, e.g. if it turns out there are legitimate, hard-to-solve edge cases. But I'm ok with that risk, as it seems highly unlikely to be problematic in practice, and there is real benefit to guaranteeing success.

Quoted 2 lines
Quoted 1 line

As for in-place conversion, this is not a blocker.

Sorry. I didn't intend to block in-place conversion.

LOL, what we intend and what happens aren't always the same. :-)

Quoted 1 line

I encountered this issue during testing, so reported it.

Thanks for doing so! I'd *much* rather sort these issues out *before* merging code, even if it means delaying the merge by a bit.

#66 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 39 lines · Sean Christopherson
Quoted 24 lines · Yan Zhao
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Rick P Edgecombe
Quoted 10 lines
Quoted 5 lines

nicely so we actually just run an old branch's TDX selftests against newer kernels. So the branch is a bit of a pile, and not really suitable for sharing. We plan to clean it and upstream it when the path clears. So it would really help to get those basic ones upstream. We remain happy to help, so please let us know.

I guess at this point I'm hoping y'all and Sean are okay that this conversions series merges, and we let this stress test failure be handled later. I'll be around to fix things :)

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'd say the line of sight to fixing this would be when the KVM MMU only gets PFNs (and no pages at all) from guest_memfd.

Hmm, I think we shouldn't upstream a uABI that we don't have line of sight to making robust. So it would be good to settle this thread at least.

This isn't uABI. You're talking about hitting a race condition between one task

Hmm. Perhaps it is not a uABI issue, since users are allowed to retry. However, it is hard to convince me that it makes sense to require users to retry a private-to-shared conversion before a GFN has ever been mapped, given that a retry is not required when the GFN is currently in use by the guest.

Quoted 5 lines
Quoted 3 lines

converting a page and another faulting in the same page. An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

Could you elaborate on how an NMI, SMI, or IRQ at just the right/wrong time could lead to the same test failures?

Ah, sorry, my bad. I was speed reading and missed that the key to your suggested "*page = NULL" change was that the reference was put _before_ filemap_invalidate_unlock_shared(), i.e. before dropping
the invalidate lock and thus before __kvm_gmem_set_attributes() will walk the folios to look for outstanding references. I was thinking that putting the reference right away was just shrinking the timing window, but putting the reference while still holding the invalidate lock closes the window entirely.

Quoted 3 lines

So, I take back what I said about this not being ABI, and about this not blocking in-place conversion. It most definitely affects ABI, and so needs to be addressed before merging in-place conversion.

I thought back then when David suggested that conversion can return -EAGAIN, one of the core ABI benefits is that this leaves the door open for things to gradually improve. If we can improve stuff within the kernel, then the the kernel would just return fewer errors. This retains backward compatibility, since extra userspace code that handles errors can continue to exist, it just won't be used.

Quoted 5 lines

The only question is if we want to commit to
guaranteeing that conversion will succeed in this scenario, or if we want to take the easy way out and formally document that conversion can fail with EAGAIN at any time, even if userspace has never mmap()'d the memory in question.

I don't really think there's a need to commit to this, IIUC in principle, ignoring that on many paths of those guest_memfd may be excluded, refcounts can be taken even if there are no host userspace mappings. For one, memory failure handling doesn't care if there are mappings, the refcount will be taken for a short while and could cause this conversion failure.

Here's the relevant part of the documentation added for conversions:

If this ioctl returns -EAGAIN, the offset of the page with unexpected refcounts will be returned in error_offset. This can occur if there are transient refcounts on the pages, taken by other parts of the kernel.

Userspace is expected to figure out how to remove all known refcounts on the shared pages, such as refcounts taken by get_user_pages(), and try the ioctl again. A possible source of these long term refcounts is if the guest_memfd memory was pinned in IOMMU page tables.

Quoted 7 lines

I'm leaning pretty strongly towards guaranteeing conversion will succeed. We'll still need to document the EAGAIN behavior, but IMO there's a massive difference between conversion failing if there's a lingering reference acquired via a VMA, conversion failing because a vCPU page fault raced with conversion. E.g. being able to assert success in a very curated test, as the stress test presumably does, would be extremely valuable for helping detect/prevent edge case bugs.

Quoted 5 lines

The argument against guaranteeing success is that we might make our future lives harder, e.g. if it turns out there are legitimate, hard-to-solve edge cases. But I'm ok with that risk, as it seems highly unlikely to be problematic in practice, and there is real benefit to guaranteeing success.

Is there really a need to commit to anything? This is already documented as "can fail", and it's orthogonal to whether the memory was mapped. The transient nature of refcounts on pages in general makes it hard to guarantee, and this stretches outside of KVM. I mean, anything could take a refcount on a page in future and we can't be auditing the entire kernel for no refcounts on guest_memfd pages ever.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

As for in-place conversion, this is not a blocker.

Sorry. I didn't intend to block in-place conversion.

LOL, what we intend and what happens aren't always the same. :-)

I don't think we're ready to guarantee conversion success when guest_memfd pages are not mapped to userspace without dragging this out way further.

I'm all for KVM not taking any references on guest_memfd, but I think eliminating KVM itself as a source of transient refcounts can be a series in itself. KVM not taking any references on guest_memfd memory is definitely welcome, it'll pave the way to using non-struct-page memory in guest_memfd.

It'll come, can we not block on this please? If we find a way to strengthen the guarantee, wouldn't that be an iterative improvement?

Quoted 4 lines
Quoted 1 line

I encountered this issue during testing, so reported it.

Thanks for doing so! I'd *much* rather sort these issues out *before* merging code, even if it means delaying the merge by a bit.

#67 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 20 lines · Ackerley Tng
Quoted 15 lines · Sean Christopherson
Quoted 5 lines · Yan Zhao
Quoted 3 lines

converting a page and another faulting in the same page. An NMI, SMI, or IRQ at just the right/wrong time, especially on a preemptible kernel, could lead to the same test failures, even if KVM drops the refcount "immediately".

Could you elaborate on how an NMI, SMI, or IRQ at just the right/wrong time could lead to the same test failures?

Ah, sorry, my bad. I was speed reading and missed that the key to your suggested "*page = NULL" change was that the reference was put _before_ filemap_invalidate_unlock_shared(), i.e. before dropping
the invalidate lock and thus before __kvm_gmem_set_attributes() will walk the folios to look for outstanding references. I was thinking that putting the reference right away was just shrinking the timing window, but putting the reference while still holding the invalidate lock closes the window entirely.

Quoted 3 lines

So, I take back what I said about this not being ABI, and about this not blocking in-place conversion. It most definitely affects ABI, and so needs to be addressed before merging in-place conversion.

Quoted 6 lines

I thought back then when David suggested that conversion can return -EAGAIN, one of the core ABI benefits is that this leaves the door open for things to gradually improve. If we can improve stuff within the kernel, then the the kernel would just return fewer errors. This retains backward compatibility, since extra userspace code that handles errors can continue to exist, it just won't be used.

Ya, that's definitely one of my hesitations to trying to guarantee success in the kernel.

Quoted 30 lines
Quoted 4 lines

The only question is if we want to commit to
guaranteeing that conversion will succeed in this scenario, or if we want to take the easy way out and formally document that conversion can fail with EAGAIN at any time, even if userspace has never mmap()'d the memory in question.

I don't really think there's a need to commit to this, IIUC in principle, ignoring that on many paths of those guest_memfd may be excluded, refcounts can be taken even if there are no host userspace mappings. For one, memory failure handling doesn't care if there are mappings, the refcount will be taken for a short while and could cause this conversion failure.

Here's the relevant part of the documentation added for conversions:

If this ioctl returns -EAGAIN, the offset of the page with unexpected refcounts will be returned in error_offset. This can occur if there are transient refcounts on the pages, taken by other parts of the kernel.

Userspace is expected to figure out how to remove all known refcounts on the shared pages, such as refcounts taken by get_user_pages(), and try the ioctl again. A possible source of these long term refcounts is if the guest_memfd memory was pinned in IOMMU page tables.

Quoted 7 lines

I'm leaning pretty strongly towards guaranteeing conversion will succeed. We'll still need to document the EAGAIN behavior, but IMO there's a massive difference between conversion failing if there's a lingering reference acquired via a VMA, conversion failing because a vCPU page fault raced with conversion. E.g. being able to assert success in a very curated test, as the stress test presumably does, would be extremely valuable for helping detect/prevent edge case bugs.

Quoted 8 lines
Quoted 5 lines

The argument against guaranteeing success is that we might make our future lives harder, e.g. if it turns out there are legitimate, hard-to-solve edge cases. But I'm ok with that risk, as it seems highly unlikely to be problematic in practice, and there is real benefit to guaranteeing success.

Is there really a need to commit to anything? This is already documented as "can fail", and it's orthogonal to whether the memory was mapped.

Yes, but the above docs also say "it's userspace's problem". Which I generally agree with, but that's not a very good story when it comes to KVM itself taking transient references, because then the answer becomes "Stop running all vCPUs", which I don't like. E.g. in a very pathological scenario, it's theoretically possible that conversion may never succeed. That's what gives me pause.

Quoted 4 lines

The transient nature of refcounts on pages in general makes it hard to guarantee, and this stretches outside of KVM. I mean, anything could take a refcount on a page in future and we can't be auditing the entire kernel for no refcounts on guest_memfd pages ever.

True, but at the same time, if there were never any VMAs then I would expect there to never be transient refcounts, modulo memory failure. And it'd be easy enough to document the memory failure angle.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

As for in-place conversion, this is not a blocker.

Sorry. I didn't intend to block in-place conversion.

LOL, what we intend and what happens aren't always the same. :-)

I don't think we're ready to guarantee conversion success when guest_memfd pages are not mapped to userspace

Yeah, that was too strong of wording on my part. The needle I was trying to thread was "conversion for this specific scenario, in a controlled environment, is guaranteed to succeed".

Quoted 5 lines

without dragging this out way further.

I'm all for KVM not taking any references on guest_memfd, but I think eliminating KVM itself as a source of transient refcounts can be a series in itself.

Yes, it would definitely be a separate mini-series.

Quoted 4 lines

KVM not taking any references on guest_memfd memory is definitely welcome, it'll pave the way to using non-struct-page memory in guest_memfd.

It'll come, can we not block on this please?

FWIW, it doesn't have to block initial merge, just the final release. E.g. even if we decide that this is a blocking issue, we can still land the in-place conversion series, so long as it's not exposed to userspace in the final release of 7.4 (or whatever kernel) without fixing the transient refcount issue.

Quoted 2 lines

If we find a way to strengthen the guarantee, wouldn't that be an iterative improvement?

Yes, but we do need to draw a line in the sand. E.g. if conversion failed 99% of the time because KVM was taking spurious references, I think we'd all agree that needs to be fixed before the code is released.

I'm still leaning towards saying this one has to be fixed, because it would give us a solid baseline from which to start, and a way to enforce it going forward (Yan's stress test). I could certinaly be convinced otherwise, though dropping the transiest reference seems straightforward enough that hopefully it's a moot point, i.e. we land both in 7.4 and don't actually have to make a decision.

#68 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 48 lines · Sean Christopherson

[...snip...]

Quoted 30 lines
Quoted 4 lines

The only question is if we want to commit to
guaranteeing that conversion will succeed in this scenario, or if we want to take the easy way out and formally document that conversion can fail with EAGAIN at any time, even if userspace has never mmap()'d the memory in question.

I don't really think there's a need to commit to this, IIUC in principle, ignoring that on many paths of those guest_memfd may be excluded, refcounts can be taken even if there are no host userspace mappings. For one, memory failure handling doesn't care if there are mappings, the refcount will be taken for a short while and could cause this conversion failure.

Here's the relevant part of the documentation added for conversions:

If this ioctl returns -EAGAIN, the offset of the page with unexpected refcounts will be returned in error_offset. This can occur if there are transient refcounts on the pages, taken by other parts of the kernel.

Userspace is expected to figure out how to remove all known refcounts on the shared pages, such as refcounts taken by get_user_pages(), and try the ioctl again. A possible source of these long term refcounts is if the guest_memfd memory was pinned in IOMMU page tables.

Quoted 7 lines

I'm leaning pretty strongly towards guaranteeing conversion will succeed. We'll still need to document the EAGAIN behavior, but IMO there's a massive difference between conversion failing if there's a lingering reference acquired via a VMA, conversion failing because a vCPU page fault raced with conversion. E.g. being able to assert success in a very curated test, as the stress test presumably does, would be extremely valuable for helping detect/prevent edge case bugs.

Quoted 8 lines
Quoted 5 lines

The argument against guaranteeing success is that we might make our future lives harder, e.g. if it turns out there are legitimate, hard-to-solve edge cases. But I'm ok with that risk, as it seems highly unlikely to be problematic in practice, and there is real benefit to guaranteeing success.

Is there really a need to commit to anything? This is already documented as "can fail", and it's orthogonal to whether the memory was mapped.

Yes, but the above docs also say "it's userspace's problem". Which I generally agree with, but that's not a very good story when it comes to KVM itself taking transient references, because then the answer becomes "Stop running all vCPUs", which I don't like. E.g. in a very pathological scenario, it's theoretically possible that conversion may never succeed. That's what gives me pause.

Quoted 9 lines
Quoted 4 lines

The transient nature of refcounts on pages in general makes it hard to guarantee, and this stretches outside of KVM. I mean, anything could take a refcount on a page in future and we can't be auditing the entire kernel for no refcounts on guest_memfd pages ever.

True, but at the same time, if there were never any VMAs then I would expect there to never be transient refcounts, modulo memory failure. And it'd be easy enough to document the memory failure angle.

I think even modulo memory failure the contract in mm for pages is that transient refcounts are allowed to be taken.

Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

As for in-place conversion, this is not a blocker.

Sorry. I didn't intend to block in-place conversion.

LOL, what we intend and what happens aren't always the same. :-)

I don't think we're ready to guarantee conversion success when guest_memfd pages are not mapped to userspace

Yeah, that was too strong of wording on my part. The needle I was trying to thread was "conversion for this specific scenario, in a controlled environment, is guaranteed to succeed".

So I think we can only specifically fix this case Yan reported.

Quoted 25 lines
Quoted 5 lines

without dragging this out way further.

I'm all for KVM not taking any references on guest_memfd, but I think eliminating KVM itself as a source of transient refcounts can be a series in itself.

Yes, it would definitely be a separate mini-series.

Quoted 4 lines

KVM not taking any references on guest_memfd memory is definitely welcome, it'll pave the way to using non-struct-page memory in guest_memfd.

It'll come, can we not block on this please?

FWIW, it doesn't have to block initial merge, just the final release. E.g. even if we decide that this is a blocking issue, we can still land the in-place conversion series, so long as it's not exposed to userspace in the final release of 7.4 (or whatever kernel) without fixing the transient refcount issue.

Quoted 2 lines

If we find a way to strengthen the guarantee, wouldn't that be an iterative improvement?

Yes, but we do need to draw a line in the sand. E.g. if conversion failed 99% of the time because KVM was taking spurious references, I think we'd all agree that needs to be fixed before the code is released.

Quoted 5 lines

I'm still leaning towards saying this one has to be fixed, because it would give us a solid baseline from which to start, and a way to enforce it going forward (Yan's stress test). I could certinaly be convinced otherwise, though dropping the transiest reference seems straightforward enough that hopefully it's a moot point, i.e. we land both in 7.4 and don't actually have to make a decision.

Sean seems confident enough that it's a small change to drop the transient reference so I went ahead to put together the series [1] so we can kick off the reviews.

I'll follow up soon with some testing and report on the other series [1]. Yan, if you could provide a Tested-by on either series it'd be great :)

Please send your reviews on [1] so we can make it for 7.4! ~7 weeks to the soft-close of 7.3-rc5!

[1] https://lore.kernel.org/all/[email protected]/

#69 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

In reply to Ackerley Tng
Quoted 27 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.

guest_memfd checks for unexpected refcounts to determine whether a page is still in use. The only expected refcounts after unmapping the range requested for conversion are those that are held by guest_memfd itself.

Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed to userspace. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.

Update documentation to document the error_offset field and the possible -EAGAIN error.

---

[...]

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 528

Quoted 11 lines

#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)

virt/kvm/guest_memfd.c +2 -0
index 3783e63476569..13c3989136f67 100644
@@ -524,8 +524,42 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
 	return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
+static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
+					    size_t nr_pages, pgoff_t *err_index)

I would focus on the "to_private" aspect or abstract it to "kvm_gmem_mem_has_unexpected_refs" or sth like that.

Quoted 8 lines
virt/kvm/guest_memfd.c +8 -0
+{
+	struct address_space *mapping = inode->i_mapping;
+	const int filemap_get_folios_refcount = 1;
+	pgoff_t last = start + nr_pages - 1;
+	struct folio_batch fbatch;
+	bool safe = true;
+	pgoff_t next;
+	int i;

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 546

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+
+	folio_batch_init(&fbatch);
+
+	next = start;
+	while (safe && filemap_get_folios(mapping, &next, last, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+
+			if (folio_ref_count(folio) !=
+			    folio_nr_pages(folio) + filemap_get_folios_refcount) {

I'd rather add a comment than have this filemap_get_folios_refcount.

/*
* We expect one reference per folio-page in the pagecache and one
* reference from filemap_get_folios().
*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

Quoted 21 lines
virt/kvm/guest_memfd.c +15 -1
+				safe = false;
+				*err_index = max(start, folio->index);
+				break;
+			}
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+
+	return safe;
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs)
+				     size_t nr_pages, uint64_t attrs,
+				     pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 584

Quoted 10 lines
virt/kvm/guest_memfd.c +5 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {

I'd add a comment here for the "why are we unmapping".

Signature · 3 lines
Cheers,

David
#70 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 40 lines · David Hildenbrand (Arm)
Quoted 27 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.

guest_memfd checks for unexpected refcounts to determine whether a page is still in use. The only expected refcounts after unmapping the range requested for conversion are those that are held by guest_memfd itself.

Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed to userspace. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.

Update documentation to document the error_offset field and the possible -EAGAIN error.

---

[...]

Quoted 9 lines

#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)

virt/kvm/guest_memfd.c +0 -0
index 3783e63476569..13c3989136f67 100644
@@ -524,8 +524,42 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
 	return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
Quoted 6 lines

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 528

Quoted 2 lines
virt/kvm/guest_memfd.c +2 -0
+static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
+					    size_t nr_pages, pgoff_t *err_index)

I would focus on the "to_private" aspect or abstract it to "kvm_gmem_mem_has_unexpected_refs" or sth like that.

Do you mean something like kvm_gmem_is_safe_for_to_private_conversion, as in that you want to emphasise that "safe" here refers to a to_private and not a to_shared conversion?

Perhaps a little ahead of its time, but later with restructuring for huge pages, we also need no additional refcounts other than gmem's own so that restructuring is safe, hence this function name was meant to extend there as well.

In this case "unexpected" (especially since the next patch adds checks for maybe dma pinned and unmapping), begs the question "unexpected in what way"?

Quoted 21 lines
Quoted 8 lines
virt/kvm/guest_memfd.c +8 -0
+{
+	struct address_space *mapping = inode->i_mapping;
+	const int filemap_get_folios_refcount = 1;
+	pgoff_t last = start + nr_pages - 1;
+	struct folio_batch fbatch;
+	bool safe = true;
+	pgoff_t next;
+	int i;

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 546

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+
+	folio_batch_init(&fbatch);
+
+	next = start;
+	while (safe && filemap_get_folios(mapping, &next, last, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+
+			if (folio_ref_count(folio) !=
+			    folio_nr_pages(folio) + filemap_get_folios_refcount) {

I'd rather add a comment than have this filemap_get_folios_refcount.

Quoted 6 lines

/*
* We expect one reference per folio-page in the pagecache and one
* reference from filemap_get_folios().
*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

This comment explains what's "unexpected". I can do this and switch it to kvm_gmem_mem_has_unexpected_refs() unless people have other suggestions.

I wish there was a folio_pagecache_refs(folio) that folio_expected_ref_count() can share with this, and also folio_swapcache_refs(), to solidify the definition of refcounts taken by the pagecache.

Quoted 22 lines
Quoted 22 lines
virt/kvm/guest_memfd.c +15 -1
+				safe = false;
+				*err_index = max(start, folio->index);
+				break;
+			}
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+
+	return safe;
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs)
+				     size_t nr_pages, uint64_t attrs,
+				     pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
Quoted 12 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 584

Quoted 9 lines
virt/kvm/guest_memfd.c +5 -1
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {

I'd add a comment here for the "why are we unmapping".

Does this sound right:

Unmap here to ensure that userspace page tables have no mappings, which also ensures refcounts from those mappings are dropped.

Quoted 4 lines

--
Cheers,

David

#71 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 42 lines · Ackerley Tng
Quoted 40 lines · David Hildenbrand (Arm)
Quoted 27 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory.

guest_memfd checks for unexpected refcounts to determine whether a page is still in use. The only expected refcounts after unmapping the range requested for conversion are those that are held by guest_memfd itself.

Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed to userspace. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully.

Update documentation to document the error_offset field and the possible -EAGAIN error.

---

[...]

Quoted 9 lines

#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)

virt/kvm/guest_memfd.c +0 -0
index 3783e63476569..13c3989136f67 100644
@@ -524,8 +524,42 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
 	return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
Quoted 5 lines
Quoted 5 lines

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 528

Quoted 2 lines
virt/kvm/guest_memfd.c +2 -0
+static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
+					    size_t nr_pages, pgoff_t *err_index)

I would focus on the "to_private" aspect or abstract it to "kvm_gmem_mem_has_unexpected_refs" or sth like that.

+1. Maybe "kvm_gmem_page_has_outstanding_references"?

Quoted 3 lines

Do you mean something like kvm_gmem_is_safe_for_to_private_conversion, as in that you want to emphasise that "safe" here refers to a to_private and not a to_shared conversion?

I'm obviously not David, but for me, the problem with names like kvm_gmem_is_safe_for_conversion() is that (a) it conflates what the function is literally doing with how the function is being used, which often makes the code harder to understand as it obfuscates things, and (b) can become stale or even outright broken far too easily. E.g. if KVM adds more checks on whether or not a conversion is "safe", then the name of the function is a lie because it doesn't actually check that the target data is safe for conversion, only that its "safe" for a specific aspect of conversion.

And there is real risk to hiding what a function does. E.g. looking at this code without diving into the details:

        if (to_private) {
                unmap_mapping_pages(mapping, start, nr_pages, false);

                if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
                                                     err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }
        }

and one might thing that it's perfectly find to check for "safety" before
unmapping pages. In fact, looking at the code without a priori knowledge of
the rules, and the above flat out looks wrong. E.g. I could definitely see someone "fixing" the code to:

        if (to_private) {
                if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
                                                     err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }

                unmap_mapping_pages(mapping, start, nr_pages, false);
	}

Whereas this:

        if (to_private) {
                unmap_mapping_pages(mapping, start, nr_pages, false);

                if (kvm_gmem_has_outstanding_references(inode, start, nr_pages,
							err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }
        }

helps the reader understand what's being checked without having to look at the details, and also helps communicate the ordering dependency without needing a comment.

There are definitely times where the usage of a function bleeds into its name, but usually that's because the name and the usage are on and the same. E.g. get_user() describes both the usage and the "what". And it's easy/possible to go too far in the opposite direction, e.g. by giving a play-by-play of what a function is doing, but that's why we have bikshedding sessions :-)

Quoted 1 line

Perhaps a little ahead of its time,

Ya.

Quoted 3 lines

but later with restructuring for huge pages, we also need no additional refcounts other than gmem's own so that restructuring is safe, hence this function name was meant to extend there as well.

Given that I've read that at least five times and still don't understand the nuance, I think it's safe (ha!) to say we'll need to revisit and review those changes no matter what. :-)

Quoted 3 lines

In this case "unexpected" (especially since the next patch adds checks for maybe dma pinned and unmapping), begs the question "unexpected in what way"?

Ya, that's why I like "outstanding", it succinctly captures that one or more references have been "loaned" but not yet "repaid".

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11 -0
+	struct address_space *mapping = inode->i_mapping;
+	const int filemap_get_folios_refcount = 1;
+	pgoff_t last = start + nr_pages - 1;
+	struct folio_batch fbatch;
+	bool safe = true;
+	pgoff_t next;
+	int i;
+
+	folio_batch_init(&fbatch);
+
+	next = start;
Quoted 8 lines
Quoted 8 lines

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 546

Quoted 6 lines
virt/kvm/guest_memfd.c +6 -0
+	while (safe && filemap_get_folios(mapping, &next, last, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+
+			if (folio_ref_count(folio) !=
+			    folio_nr_pages(folio) + filemap_get_folios_refcount) {

I'd rather add a comment than have this filemap_get_folios_refcount.

+1, the local variable just made me scratch my head.

Quoted 4 lines
Quoted 4 lines

/*
* We expect one reference per folio-page in the pagecache and one
* reference from filemap_get_folios().

Nit, please no pronouns in KVM code.

Quoted 12 lines
Quoted 3 lines

*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

This comment explains what's "unexpected". I can do this and switch it to kvm_gmem_mem_has_unexpected_refs() unless people have other suggestions.

I wish there was a folio_pagecache_refs(folio) that folio_expected_ref_count() can share with this, and also folio_swapcache_refs(), to solidify the definition of refcounts taken by the pagecache.

...

Quoted 15 lines
Quoted 14 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 584

Quoted 11 lines
virt/kvm/guest_memfd.c +5 -1
@@ -542,8 +576,21 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r)
+	if (r) {
+		*err_index = start;
 		goto out;
+	}
+
+	if (to_private) {

I'd add a comment here for the "why are we unmapping".

Quoted 4 lines

Does this sound right:

Unmap here to ensure that userspace page tables have no mappings, which also ensures refcounts from those mappings are dropped.

How about:

		/*
		 * Forcefully unmap the pages from all userspace page tables,
		 * and then verify there are no outstanding references, e.g.
		 * acquired via GUP or similar.  Tell userspace to try again if
		 * there are oustanding references and hope that whatever has
		 * pinned the page will put its reference "soon".
		 */
		unmap_mapping_pages(mapping, start, nr_pages, false);

                if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
                                                     err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }
Quoted 5 lines
Quoted 5 lines

--
Cheers,

David

#72 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 9 lines

helps the reader understand what's being checked without having to look at the details, and also helps communicate the ordering dependency without needing a comment.

There are definitely times where the usage of a function bleeds into its name, but usually that's because the name and the usage are on and the same. E.g. get_user() describes both the usage and the "what". And it's easy/possible to go too far in the opposite direction, e.g. by giving a play-by-play of what a function is doing, but that's why we have bikshedding sessions :-)

Note that the problem I have with kvm_gmem_is_safe_for_conversion() that it is all about *conversion to private*, not *conversion to shared*. In that sense, the function name is just confusing.

Quoted 24 lines
Quoted 2 lines

Perhaps a little ahead of its time,

Ya.

Quoted 3 lines

but later with restructuring for huge pages, we also need no additional refcounts other than gmem's own so that restructuring is safe, hence this function name was meant to extend there as well.

Given that I've read that at least five times and still don't understand the nuance, I think it's safe (ha!) to say we'll need to revisit and review those changes no matter what. :-)

Quoted 3 lines

In this case "unexpected" (especially since the next patch adds checks for maybe dma pinned and unmapping), begs the question "unexpected in what way"?

Ya, that's why I like "outstanding", it succinctly captures that one or more references have been "loaned" but not yet "repaid".

Quoted 2 lines
Quoted 2 lines

I'd rather add a comment than have this filemap_get_folios_refcount.

Quoted 8 lines

+1, the local variable just made me scratch my head.

Quoted 4 lines
Quoted 4 lines

/*
* We expect one reference per folio-page in the pagecache and one
* reference from filemap_get_folios().

Nit, please no pronouns in KVM code.

Whatever floats KVM's boat :)

Quoted 27 lines
Quoted 13 lines
Quoted 4 lines

*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

This comment explains what's "unexpected". I can do this and switch it to kvm_gmem_mem_has_unexpected_refs() unless people have other suggestions.

I wish there was a folio_pagecache_refs(folio) that folio_expected_ref_count() can share with this, and also folio_swapcache_refs(), to solidify the definition of refcounts taken by the pagecache.

...

Quoted 8 lines
Quoted 3 lines

I'd add a comment here for the "why are we unmapping".

Does this sound right:

Unmap here to ensure that userspace page tables have no mappings, which also ensures refcounts from those mappings are dropped.

How about:

Quoted 15 lines
		/*
		 * Forcefully unmap the pages from all userspace page tables,
		 * and then verify there are no outstanding references, e.g.
		 * acquired via GUP or similar.  Tell userspace to try again if
		 * there are oustanding references and hope that whatever has
		 * pinned the page will put its reference "soon".
		 */
		unmap_mapping_pages(mapping, start, nr_pages, false);

                if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
                                                     err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }

Sounds good besides the function still not being clear about *which* kind of conversion.

Signature · 3 lines
Cheers,

David
#73 lore raw

Re: [PATCH v10 11/41] KVM: guest_memfd: Ensure pages are not in use before conversion

Quoted 93 lines · David Hildenbrand (Arm)
Quoted 9 lines

helps the reader understand what's being checked without having to look at the details, and also helps communicate the ordering dependency without needing a comment.

There are definitely times where the usage of a function bleeds into its name, but usually that's because the name and the usage are on and the same. E.g. get_user() describes both the usage and the "what". And it's easy/possible to go too far in the opposite direction, e.g. by giving a play-by-play of what a function is doing, but that's why we have bikshedding sessions :-)

Note that the problem I have with kvm_gmem_is_safe_for_conversion() that it is all about *conversion to private*, not *conversion to shared*. In that sense, the function name is just confusing.

Quoted 24 lines
Quoted 2 lines

Perhaps a little ahead of its time,

Ya.

Quoted 3 lines

but later with restructuring for huge pages, we also need no additional refcounts other than gmem's own so that restructuring is safe, hence this function name was meant to extend there as well.

Given that I've read that at least five times and still don't understand the nuance, I think it's safe (ha!) to say we'll need to revisit and review those changes no matter what. :-)

Quoted 3 lines

In this case "unexpected" (especially since the next patch adds checks for maybe dma pinned and unmapping), begs the question "unexpected in what way"?

Ya, that's why I like "outstanding", it succinctly captures that one or more references have been "loaned" but not yet "repaid".

Quoted 2 lines
Quoted 2 lines

I'd rather add a comment than have this filemap_get_folios_refcount.

Quoted 8 lines

+1, the local variable just made me scratch my head.

Quoted 4 lines
Quoted 4 lines

/*
* We expect one reference per folio-page in the pagecache and one
* reference from filemap_get_folios().

Nit, please no pronouns in KVM code.

Whatever floats KVM's boat :)

Quoted 27 lines
Quoted 13 lines
Quoted 4 lines

*/
if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

This comment explains what's "unexpected". I can do this and switch it to kvm_gmem_mem_has_unexpected_refs() unless people have other suggestions.

I wish there was a folio_pagecache_refs(folio) that folio_expected_ref_count() can share with this, and also folio_swapcache_refs(), to solidify the definition of refcounts taken by the pagecache.

...

Quoted 8 lines
Quoted 3 lines

I'd add a comment here for the "why are we unmapping".

Does this sound right:

Unmap here to ensure that userspace page tables have no mappings, which also ensures refcounts from those mappings are dropped.

How about:

Quoted 15 lines
		/*
		 * Forcefully unmap the pages from all userspace page tables,
		 * and then verify there are no outstanding references, e.g.
		 * acquired via GUP or similar.  Tell userspace to try again if
		 * there are oustanding references and hope that whatever has
		 * pinned the page will put its reference "soon".
		 */
		unmap_mapping_pages(mapping, start, nr_pages, false);

                if (!kvm_gmem_is_safe_for_conversion(inode, start, nr_pages,
                                                     err_index)) {
                        mas_destroy(&mas);
                        r = -EAGAIN;
                        goto out;
                }
Quoted 3 lines

Sounds good besides the function still not being clear about *which* kind of conversion.

Circling back to this since Binbin also had a comment about the comment!

I'll drop the use of the const local variable and do

    if (folio_ref_count(folio) != folio_nr_pages(folio) + 1)

, and add a comment rephrased without the pronouns:

No references outstanding if guest_memfd sees 1 reference per folio-page in the pagecache and 1 reference from filemap_get_folios()

, and then also take the comment and rename the function from kvm_gmem_is_safe_for_conversion to kvm_gmem_has_outstanding_references.

	if (to_private) {
		/*
		 * Forcefully unmap the pages from all userspace page tables,
		 * and then verify there are no outstanding references, e.g.
		 * acquired via GUP or similar.	 Tell userspace to try again if
		 * there are oustanding references and hope that whatever has
		 * pinned the page will put its reference "soon".
		 */
		 unmap_mapping_pages(mapping, start, nr_pages, false);

		 if (kvm_gmem_has_outstanding_references(inode, start, nr_pages,
							 err_index)) {
			mas_destroy(&mas);
			r = -EAGAIN;
			goto out;
		 }
	}

I think renaming the function from kvm_gmem_is_safe_for_conversion to kvm_gmem_has_outstanding_references should be better since it doesn't really make a statement on what kind of conversion the outstanding references are for.

Quoted 5 lines

--
Cheers,

David

#74 lore raw

PATCH v10 12/41 KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

In reply to Ackerley Tng

When memory in guest_memfd is converted from private to shared, the platform-specific state associated with the guest-private pages must be invalidated or cleaned up.

Iterate over the folios in the affected range and call the kvm_arch_gmem_make_shared() hook for each PFN range. This allows architectures to update hardware metadata or encryption states to transition pages to the shared state.

Invoke this helper after indicating to KVM's mmu code that an invalidation is in progress to stop in-flight page faults from succeeding.

Omit support for calling the arch hook to make private, since SNP, the only implementer of the arch make-private hook today, would actually prefer making private only just before faulting memory into the NPTs.

Calling the make-private arch hook would require iterating both bindings and the filemap to find the intersection of bindings and allocated folios. On top of that, SNP would need to figure out whether to actually make private based on whether the memory is about the be faulted, or whether it is a conversion.

This does leak SNP-specific details into guest_memfd (as in, why only make-shared during conversions but not make-private?), but the additional complexity is not worth taking on until guest_memfd has a user actually requiring an arch make-private call.

---
 arch/x86/include/asm/kvm-x86-ops.h |  2 +-
 arch/x86/include/asm/kvm_host.h    |  2 +-
 arch/x86/kvm/x86.c                 |  5 +++++
 include/linux/kvm_host.h           |  1 +
 virt/kvm/guest_memfd.c             | 42 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 50 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
index e213c9ae3e301..67b43c167045b 100644
@@ -150,7 +150,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 KVM_X86_OP_OPTIONAL_RET0(gmem_make_private)
 #endif
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 KVM_X86_OP_OPTIONAL(gmem_make_shared)
 #endif
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/include/asm/kvm_host.h +1 -1
index 14d86851fce78..497da6aaf2942 100644
@@ -1732,7 +1732,7 @@ struct kvm_x86_ops {
 	int (*gmem_make_private)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 				 kvm_pfn_t nr_pages);
 #endif
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 	void (*gmem_make_shared)(kvm_pfn_t pfn, kvm_pfn_t nr_pages);
 #endif
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/x86.c +5 -0
index f1d977060489b..39968610868fd 100644
@@ -10647,6 +10647,11 @@ int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 {
 	return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages);
 }
+
+void kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages)
+{
+	kvm_x86_call(gmem_make_shared)(pfn, nr_pages);
+}
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
include/linux/kvm_host.h +1 -0
index dfdb3c97a7ac9..a341c434a2b7e 100644
@@ -2613,6 +2613,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			       kvm_pfn_t nr_pages);
+void kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +42 -0

55 lines not on this page · raw message

Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#75 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 14 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

When memory in guest_memfd is converted from private to shared, the platform-specific state associated with the guest-private pages must be invalidated or cleaned up.

Iterate over the folios in the affected range and call the kvm_arch_gmem_make_shared() hook for each PFN range. This allows architectures to update hardware metadata or encryption states to transition pages to the shared state.

Invoke this helper after indicating to KVM's mmu code that an invalidation is in progress to stop in-flight page faults from succeeding.

Quoted 8 lines

Omit support for calling the arch hook to make private, since SNP, the only implementer of the arch make-private hook today, would actually prefer making private only just before faulting memory into the NPTs.

Calling the make-private arch hook would require iterating both bindings and the filemap to find the intersection of bindings and allocated folios. On top of that, SNP would need to figure out whether to actually make private based on whether the memory is about the be faulted, or

                                                     ^

the -> to?

Quoted 2 lines

whether it is a conversion.

[...]

Quoted 42 lines
virt/kvm/guest_memfd.c +37 -0
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+			pgoff_t start_index, end_index;
+			kvm_pfn_t start_pfn;
+			kvm_pfn_t nr_pages;
+
+			start_index = max(start, folio->index);
+			end_index = min(end, folio_next_index(folio));
+			/*
+			 * end_index is either in folio or points to
+			 * the first page of the next folio. Hence,
+			 * all pages in range [start_index, end_index)
+			 * are contiguous.
+			 */
+			start_pfn = folio_file_pfn(folio, start_index);
+			nr_pages = end_index - start_index;
+
+			kvm_arch_gmem_make_shared(start_pfn, nr_pages);
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+}
+#else
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+#endif
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 641

Quoted 6 lines
virt/kvm/guest_memfd.c +3 -0
@@ -599,7 +636,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
+
+	if (!to_private)
+		kvm_gmem_make_shared(inode, start, end);

If both KVM_AMD_SEV and KVM_INTEL_TDX are enabled, HAVE_KVM_ARCH_GMEM_CONVERT will be enabled and the logic in kvm_gmem_make_shared() introduces unnecessary overhead for TDX.
Not sure about CSPs, but in a standard distribution kernel, it's very likely that both are enabled, right? Should kvm_gmem_make_shared() do some optimization or the overhead is relative small in the conversion to shared path so that the optimization is not worth it?

Quoted 7 lines
virt/kvm/guest_memfd.c +2 -0
+
 	mas_store_prealloc(&mas, xa_mk_value(attrs));
+
 	kvm_gmem_invalidate_end(inode, start, end);
 out:
 	filemap_invalidate_unlock(mapping);
#76 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 43 lines · Binbin Wu
Quoted 42 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:
virt/kvm/guest_memfd.c +37 -0
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+			pgoff_t start_index, end_index;
+			kvm_pfn_t start_pfn;
+			kvm_pfn_t nr_pages;
+
+			start_index = max(start, folio->index);
+			end_index = min(end, folio_next_index(folio));
+			/*
+			 * end_index is either in folio or points to
+			 * the first page of the next folio. Hence,
+			 * all pages in range [start_index, end_index)
+			 * are contiguous.
+			 */
+			start_pfn = folio_file_pfn(folio, start_index);
+			nr_pages = end_index - start_index;
+
+			kvm_arch_gmem_make_shared(start_pfn, nr_pages);
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+}
+#else
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+#endif
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -599,7 +636,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
Quoted 11 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 641

Quoted 5 lines
virt/kvm/guest_memfd.c +3 -0
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
+
+	if (!to_private)
+		kvm_gmem_make_shared(inode, start, end);

If both KVM_AMD_SEV and KVM_INTEL_TDX are enabled, HAVE_KVM_ARCH_GMEM_CONVERT will be enabled and the logic in kvm_gmem_make_shared() introduces unnecessary overhead for TDX.
Not sure about CSPs, but in a standard distribution kernel, it's very likely that both are enabled, right?

Yep.

Quoted 2 lines

Should kvm_gmem_make_shared() do some optimization or the overhead is relative small in the conversion to shared path so that the optimization is not worth it?

I assume it's relatively small, but it's probably a good idea to gets numbers so that we can make a semi-informed decision. E.g. if processing 1GiB work of 4KiB folios is <100 cycles, I think we generally don't care. But if it's more like tens of microseconds, then I think we do care enough to optimize the walking for TDX.

#77 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 44 lines · Sean Christopherson
Quoted 43 lines · Binbin Wu
Quoted 42 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:
virt/kvm/guest_memfd.c +37 -0
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+			pgoff_t start_index, end_index;
+			kvm_pfn_t start_pfn;
+			kvm_pfn_t nr_pages;
+
+			start_index = max(start, folio->index);
+			end_index = min(end, folio_next_index(folio));
+			/*
+			 * end_index is either in folio or points to
+			 * the first page of the next folio. Hence,
+			 * all pages in range [start_index, end_index)
+			 * are contiguous.
+			 */
+			start_pfn = folio_file_pfn(folio, start_index);
+			nr_pages = end_index - start_index;
+
+			kvm_arch_gmem_make_shared(start_pfn, nr_pages);
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+}
+#else
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+#endif
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -599,7 +636,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
Quoted 4 lines
Quoted 4 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 640

Quoted 4 lines
virt/kvm/guest_memfd.c +2 -0
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
+
+	if (!to_private)

Should I make this condition

	if (!to_private && kvm_x86_ops.gmem_make_shared)

instead?

Will that help?

Quoted 18 lines
Quoted 7 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 641

Quoted 1 line
virt/kvm/guest_memfd.c +1 -0
+		kvm_gmem_make_shared(inode, start, end);

If both KVM_AMD_SEV and KVM_INTEL_TDX are enabled, HAVE_KVM_ARCH_GMEM_CONVERT will be enabled and the logic in kvm_gmem_make_shared() introduces unnecessary overhead for TDX.
Not sure about CSPs, but in a standard distribution kernel, it's very likely that both are enabled, right?

Yep.

Quoted 2 lines

Should kvm_gmem_make_shared() do some optimization or the overhead is relative small in the conversion to shared path so that the optimization is not worth it?

I assume it's relatively small, but it's probably a good idea to gets numbers so that we can make a semi-informed decision. E.g. if processing 1GiB work of 4KiB folios is <100 cycles, I think we generally don't care. But if it's more like tens of microseconds, then I think we do care enough to optimize the walking for TDX.

#78 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 53 lines · Ackerley Tng
Quoted 44 lines · Sean Christopherson
Quoted 43 lines · Binbin Wu
Quoted 42 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:
virt/kvm/guest_memfd.c +37 -0
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+			pgoff_t start_index, end_index;
+			kvm_pfn_t start_pfn;
+			kvm_pfn_t nr_pages;
+
+			start_index = max(start, folio->index);
+			end_index = min(end, folio_next_index(folio));
+			/*
+			 * end_index is either in folio or points to
+			 * the first page of the next folio. Hence,
+			 * all pages in range [start_index, end_index)
+			 * are contiguous.
+			 */
+			start_pfn = folio_file_pfn(folio, start_index);
+			nr_pages = end_index - start_index;
+
+			kvm_arch_gmem_make_shared(start_pfn, nr_pages);
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+}
+#else
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+#endif
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -599,7 +636,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
Quoted 4 lines
Quoted 4 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 640

Quoted 4 lines
virt/kvm/guest_memfd.c +2 -0
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
+
+	if (!to_private)

Should I make this condition

Quoted 5 lines
	if (!to_private && kvm_x86_ops.gmem_make_shared)

instead?

Will that help?

No, because that's bleeding x86 details into common code, which defeats the purpose of arch hooks. That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

#79 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 54 lines · Sean Christopherson
Quoted 53 lines · Ackerley Tng
Quoted 44 lines · Sean Christopherson
Quoted 43 lines · Binbin Wu
Quoted 42 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:
virt/kvm/guest_memfd.c +37 -0
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
+			struct folio *folio = fbatch.folios[i];
+			pgoff_t start_index, end_index;
+			kvm_pfn_t start_pfn;
+			kvm_pfn_t nr_pages;
+
+			start_index = max(start, folio->index);
+			end_index = min(end, folio_next_index(folio));
+			/*
+			 * end_index is either in folio or points to
+			 * the first page of the next folio. Hence,
+			 * all pages in range [start_index, end_index)
+			 * are contiguous.
+			 */
+			start_pfn = folio_file_pfn(folio, start_index);
+			nr_pages = end_index - start_index;
+
+			kvm_arch_gmem_make_shared(start_pfn, nr_pages);
+		}
+
+		folio_batch_release(&fbatch);
+		cond_resched();
+	}
+}
+#else
+static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+#endif
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -599,7 +636,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
Quoted 4 lines
Quoted 4 lines

virt/kvm/guest_memfd.c static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, · line 640

Quoted 4 lines
virt/kvm/guest_memfd.c +2 -0
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
+
+	if (!to_private)

Should I make this condition

Quoted 8 lines
Quoted 5 lines
	if (!to_private && kvm_x86_ops.gmem_make_shared)

instead?

Will that help?

No, because that's bleeding x86 details into common code, which defeats the purpose of arch hooks.

Ah okay yup maybe it should have been kvm_arch_has_gmem_make_shared() that defaults to false and x86 will return !!kvm_x86_ops.gmem_make_shared.

Quoted 3 lines

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Is noise defined relative to the entire conversion process? Would this benchmark look like

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

#80 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Quoted 7 lines

Is noise defined relative to the entire conversion process? Would this benchmark look like

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

That'd work, though I was envisioning something even simpler: use rdtsc() to count the cycles it takes to iterate over various ranges of memory. Do whatever is easiest for you though.

#81 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Is noise defined relative to the entire conversion process? Would this benchmark look like

Quoted 8 lines
Quoted 4 lines

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

That'd work, though I was envisioning something even simpler: use rdtsc() to count the cycles it takes to iterate over various ranges of memory. Do whatever is easiest for you though.

I made some changes to add rdtsc() for the conversion process as Sean suggested [1], and exercised conversion like this [2]:

1. Initialize some memory as private
2. Get the guest to fault them into Secure EPTs
3. Converts the memory to shared <<== this is being benchmarked
4. Converts memory back to private

I made it build the VM once and convert 5 times:

./gmem_benchmark_tdx_convert --iterations=5 --size=1g

Here are the raw prints:

[  396.403818] guest_memfd to_shared: nr_pages=1, make_shared=930
cycles, total=39278 cycles
[  396.412087] guest_memfd to_shared: nr_pages=1, make_shared=252
cycles, total=28060 cycles
[  396.420379] guest_memfd to_shared: nr_pages=1, make_shared=176
cycles, total=26952 cycles
[  396.428643] guest_memfd to_shared: nr_pages=1, make_shared=176
cycles, total=27038 cycles
[  396.436921] guest_memfd to_shared: nr_pages=1, make_shared=176

cycles, total=26980 cycles

[  400.338157] guest_memfd to_shared: nr_pages=1, make_shared=1072
cycles, total=37236 cycles
[  400.346509] guest_memfd to_shared: nr_pages=1, make_shared=316
cycles, total=28338 cycles
[  400.354805] guest_memfd to_shared: nr_pages=1, make_shared=176
cycles, total=27182 cycles
[  400.363090] guest_memfd to_shared: nr_pages=1, make_shared=176
cycles, total=26972 cycles
[  400.371349] guest_memfd to_shared: nr_pages=1, make_shared=176

cycles, total=26886 cycles

[  448.761474] guest_memfd to_shared: nr_pages=262144,
make_shared=15041018 cycles, total=6616067680 cycles
[  455.807988] guest_memfd to_shared: nr_pages=262144,
make_shared=14937462 cycles, total=6608542680 cycles
[  462.848856] guest_memfd to_shared: nr_pages=262144,
make_shared=15138858 cycles, total=6599494898 cycles
[  469.885694] guest_memfd to_shared: nr_pages=262144,
make_shared=15721972 cycles, total=6610219850 cycles
[  476.928399] guest_memfd to_shared: nr_pages=262144,

make_shared=15000406 cycles, total=6615114540 cycles

[  762.980690] guest_memfd to_shared: nr_pages=1048576,
make_shared=61902982 cycles, total=26400884028 cycles
[  791.022885] guest_memfd to_shared: nr_pages=1048576,
make_shared=61746114 cycles, total=26401170984 cycles
[  819.054580] guest_memfd to_shared: nr_pages=1048576,
make_shared=61096794 cycles, total=26404409058 cycles
[  847.120994] guest_memfd to_shared: nr_pages=1048576,
make_shared=61446290 cycles, total=26447461896 cycles
[  875.219750] guest_memfd to_shared: nr_pages=1048576,

make_shared=61774646 cycles, total=26444608360 cycles

And here's the above, tabulated:

nr_pages make_shared total percentage
---------- --------------- --------------- ------------

         1              930            39278       2.3677%
         1              252            28060       0.8981%
         1              176            26952       0.6530%
         1              176            27038       0.6509%
         1              176            26980       0.6523%
         1             1072            37236       2.8789%
         1              316            28338       1.1151%
         1              176            27182       0.6475%
         1              176            26972       0.6525%
         1              176            26886       0.6546%
    262144         15041018       6616067680       0.2273%
    262144         14937462       6608542680       0.2260%
    262144         15138858       6599494898       0.2294%
    262144         15721972       6610219850       0.2378%
    262144         15000406       6615114540       0.2268%
   1048576         61902982      26400884028       0.2345%
   1048576         61746114      26401170984       0.2339%
   1048576         61096794      26404409058       0.2314%
   1048576         61446290      26447461896       0.2323%
   1048576         61774646      26444608360       0.2336%

Looks to me it is within noise.

I also actually tried measuring the conversion time from userspace with CONFIG_AMD_SEV enabled and disabled. Converting a 1G-sized TD was faster by 0.2%, which is in line with the above table. Interestingly, when converting a 4G-sized TD, skipping kvm_gmem_make_shared() was _slower_ over 2 runs. I don't have an explanation for that. I think the code was correct. (If it makes a difference, I skipped kvm_gmem_make_shared() using a custom guest_memfd creation time flag and skipped make_shared if the flag was set on the inode.)

I thought adding a kvm_arch_has_gmem_make_shared(), defaulting it to false for all archs and having x86 override with !!kvm_x86_ops.gmem_make_shared is not too bad either:

+ doesn't leak anything, since the function being called is kvm_arch_gmem_make_shared and the accompanying function is kvm_arch_has_gmem_make_shared. Or maybe just a little, since all the other ops don't have the accompanying _has_ function
+ it's a kernel-internal thing
+ not too many lines of code, not too complex

Sean, let me know if you prefer to add kvm_arch_has_gmem_make_shared() and for my learning, what your considerations are regarding adding kvm_arch_has_gmem_make_shared() or not.

[1] https://github.com/googleprodkernel/linux-cc/commit/3295072cfeb8920e48d9e80c98fd1695d4bbad99
[2] https://github.com/googleprodkernel/linux-cc/commit/a644618f270f02c832fb7e8402fcae98bc91f4e5

#82 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 31 lines · Ackerley Tng
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Is noise defined relative to the entire conversion process? Would this benchmark look like

Quoted 8 lines
Quoted 4 lines

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

That'd work, though I was envisioning something even simpler: use rdtsc() to count the cycles it takes to iterate over various ranges of memory. Do whatever is easiest for you though.

I made some changes to add rdtsc() for the conversion process as Sean suggested [1], and exercised conversion like this [2]:

1. Initialize some memory as private
2. Get the guest to fault them into Secure EPTs
3. Converts the memory to shared <<== this is being benchmarked
4. Converts memory back to private

I made it build the VM once and convert 5 times:

./gmem_benchmark_tdx_convert --iterations=5 --size=1g

...

Quoted 25 lines

And here's the above, tabulated:

nr_pages make_shared total percentage
---------- --------------- --------------- ------------

         1              930            39278       2.3677%
         1              252            28060       0.8981%
         1              176            26952       0.6530%
         1              176            27038       0.6509%
         1              176            26980       0.6523%
         1             1072            37236       2.8789%
         1              316            28338       1.1151%
         1              176            27182       0.6475%
         1              176            26972       0.6525%
         1              176            26886       0.6546%
    262144         15041018       6616067680       0.2273%
    262144         14937462       6608542680       0.2260%
    262144         15138858       6599494898       0.2294%
    262144         15721972       6610219850       0.2378%
    262144         15000406       6615114540       0.2268%
   1048576         61902982      26400884028       0.2345%
   1048576         61746114      26401170984       0.2339%
   1048576         61096794      26404409058       0.2314%
   1048576         61446290      26447461896       0.2323%
   1048576         61774646      26444608360       0.2336%
Quoted 7 lines

Looks to me it is within noise.

I also actually tried measuring the conversion time from userspace with CONFIG_AMD_SEV enabled and disabled. Converting a 1G-sized TD was faster by 0.2%, which is in line with the above table. Interestingly, when converting a 4G-sized TD, skipping kvm_gmem_make_shared() was _slower_ over 2 runs. I don't have an explanation for that.

Might be some cache/memory locality benefits? Though with a conversion that big, it could also be nothing more than bad luck.

Quoted 5 lines

I think the code was correct. (If it makes a difference, I skipped kvm_gmem_make_shared() using a custom guest_memfd creation time flag and skipped make_shared if the flag was set on the inode.)

I thought adding a kvm_arch_has_gmem_make_shared(), defaulting it to

I would do kvm_arch_has_gmem_convert() for consistency with the Kconfigs, and because the cost of the reclaim invocation is a non-issue.

Quoted 8 lines

false for all archs and having x86 override with !!kvm_x86_ops.gmem_make_shared is not too bad either:
+ doesn't leak anything, since the function being called is kvm_arch_gmem_make_shared and the accompanying function is kvm_arch_has_gmem_make_shared. Or maybe just a little, since all the other ops don't have the accompanying _has_ function
+ it's a kernel-internal thing
+ not too many lines of code, not too complex

It also provides a good excuse to kill off the #idfefs in guest_memfd.c. Compile tested only, but I'm thinking this?

From: Sean Christopherson <[email protected]>
Date: Mon, 17 Aug 2026 12:31:50 -0700
Subject: [PATCH] KVM: guest_memfd: Optimize away conversion overheads via
dead-code elimination

Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleed guest_memfd details into arch code and would be far worse than adding yet another kvm_arch_has... hook.

Opportunistically provide the kvm_arch_gmem_make_private() declaration, and rely on dead-code elimination to eliminate the call to non-existent code when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.

---
 arch/x86/include/asm/kvm_host.h | 3 +++
 include/linux/kvm_host.h        | 3 ++-
 virt/kvm/guest_memfd.c          | 5 ++---
 3 files changed, 7 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
index 283847619ff8..5d5a7723abb6 100644
@@ -1854,6 +1854,9 @@ enum kvm_intr_type {
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+#define kvm_arch_has_gmem_convert() (!!kvm_x86_ops.gmem_make_private)
+#endif
 
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
 
include/linux/kvm_host.h +2 -1
index 03bfc92864b6..e824ba59c60c 100644
@@ -2599,9 +2599,10 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			       kvm_pfn_t nr_pages);
+#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+#define kvm_arch_has_gmem_convert() false
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +2 -3
index b596486d184c..39d94938b5f6 100644
@@ -773,11 +773,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		folio_mark_uptodate(folio);
 	}
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
-	if (kvm_gmem_is_private_mem(file_inode(file), index))
+	if (kvm_arch_has_gmem_convert() &&
+	    kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
 					       (kvm_pfn_t)1 << *max_order);
-#endif
 
 	folio_unlock(folio);
 

base-commit: 1b731e5ded480bd1e5546aed35584238661ce72e

#83 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

Quoted 71 lines · Sean Christopherson
Quoted 31 lines · Ackerley Tng
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Is noise defined relative to the entire conversion process? Would this benchmark look like

Quoted 8 lines
Quoted 4 lines

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

That'd work, though I was envisioning something even simpler: use rdtsc() to count the cycles it takes to iterate over various ranges of memory. Do whatever is easiest for you though.

I made some changes to add rdtsc() for the conversion process as Sean suggested [1], and exercised conversion like this [2]:

1. Initialize some memory as private
2. Get the guest to fault them into Secure EPTs
3. Converts the memory to shared <<== this is being benchmarked
4. Converts memory back to private

I made it build the VM once and convert 5 times:

./gmem_benchmark_tdx_convert --iterations=5 --size=1g

...

Quoted 25 lines

And here's the above, tabulated:

nr_pages make_shared total percentage
---------- --------------- --------------- ------------

         1              930            39278       2.3677%
         1              252            28060       0.8981%
         1              176            26952       0.6530%
         1              176            27038       0.6509%
         1              176            26980       0.6523%
         1             1072            37236       2.8789%
         1              316            28338       1.1151%
         1              176            27182       0.6475%
         1              176            26972       0.6525%
         1              176            26886       0.6546%
    262144         15041018       6616067680       0.2273%
    262144         14937462       6608542680       0.2260%
    262144         15138858       6599494898       0.2294%
    262144         15721972       6610219850       0.2378%
    262144         15000406       6615114540       0.2268%
   1048576         61902982      26400884028       0.2345%
   1048576         61746114      26401170984       0.2339%
   1048576         61096794      26404409058       0.2314%
   1048576         61446290      26447461896       0.2323%
   1048576         61774646      26444608360       0.2336%
Quoted 7 lines

Looks to me it is within noise.

I also actually tried measuring the conversion time from userspace with CONFIG_AMD_SEV enabled and disabled. Converting a 1G-sized TD was faster by 0.2%, which is in line with the above table. Interestingly, when converting a 4G-sized TD, skipping kvm_gmem_make_shared() was _slower_ over 2 runs. I don't have an explanation for that.

Might be some cache/memory locality benefits? Though with a conversion that big, it could also be nothing more than bad luck.

Quoted 8 lines
Quoted 5 lines

I think the code was correct. (If it makes a difference, I skipped kvm_gmem_make_shared() using a custom guest_memfd creation time flag and skipped make_shared if the flag was set on the inode.)

I thought adding a kvm_arch_has_gmem_make_shared(), defaulting it to

I would do kvm_arch_has_gmem_convert() for consistency with the Kconfigs, and because the cost of the reclaim invocation is a non-issue.

Forgot to say "why" I think it's worth adding kvm_arch_has_gmem_convert(): it's a trivial change in the grand scheme, and I can see the ~1000 cycles of overhead being problematic in use cases that do many one-off conversions, e.g. for guests whose shared memory is scattered all over.

#84 lore raw

Re: [PATCH v10 12/41] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion

In reply to Sean Christopherson
Quoted 170 lines · Sean Christopherson
Quoted 31 lines · Ackerley Tng
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson

That's why I think it's worth analyzing the cost: if it's in the noise, leave it alone. If it's meaningful, figure out a not-too-gross way to skip the entire thing if kvm_arch_gmem_make_shared() is a glorified nop in the end.

Is noise defined relative to the entire conversion process? Would this benchmark look like

Quoted 8 lines
Quoted 4 lines

1. Convert 4G to shared on TDX with CONFIG_AMD_SEV defined
2. Convert 4G to shared on TDX without CONFIG_AMD_SEV defined

and then compare the difference in time taken?

That'd work, though I was envisioning something even simpler: use rdtsc() to count the cycles it takes to iterate over various ranges of memory. Do whatever is easiest for you though.

I made some changes to add rdtsc() for the conversion process as Sean suggested [1], and exercised conversion like this [2]:

1. Initialize some memory as private
2. Get the guest to fault them into Secure EPTs
3. Converts the memory to shared <<== this is being benchmarked
4. Converts memory back to private

I made it build the VM once and convert 5 times:

./gmem_benchmark_tdx_convert --iterations=5 --size=1g

...

Quoted 25 lines

And here's the above, tabulated:

nr_pages make_shared total percentage
---------- --------------- --------------- ------------

         1              930            39278       2.3677%
         1              252            28060       0.8981%
         1              176            26952       0.6530%
         1              176            27038       0.6509%
         1              176            26980       0.6523%
         1             1072            37236       2.8789%
         1              316            28338       1.1151%
         1              176            27182       0.6475%
         1              176            26972       0.6525%
         1              176            26886       0.6546%
    262144         15041018       6616067680       0.2273%
    262144         14937462       6608542680       0.2260%
    262144         15138858       6599494898       0.2294%
    262144         15721972       6610219850       0.2378%
    262144         15000406       6615114540       0.2268%
   1048576         61902982      26400884028       0.2345%
   1048576         61746114      26401170984       0.2339%
   1048576         61096794      26404409058       0.2314%
   1048576         61446290      26447461896       0.2323%
   1048576         61774646      26444608360       0.2336%
Quoted 7 lines

Looks to me it is within noise.

I also actually tried measuring the conversion time from userspace with CONFIG_AMD_SEV enabled and disabled. Converting a 1G-sized TD was faster by 0.2%, which is in line with the above table. Interestingly, when converting a 4G-sized TD, skipping kvm_gmem_make_shared() was _slower_ over 2 runs. I don't have an explanation for that.

Might be some cache/memory locality benefits? Though with a conversion that big, it could also be nothing more than bad luck.

Quoted 5 lines

I think the code was correct. (If it makes a difference, I skipped kvm_gmem_make_shared() using a custom guest_memfd creation time flag and skipped make_shared if the flag was set on the inode.)

I thought adding a kvm_arch_has_gmem_make_shared(), defaulting it to

I would do kvm_arch_has_gmem_convert() for consistency with the Kconfigs, and because the cost of the reclaim invocation is a non-issue.

Quoted 8 lines

false for all archs and having x86 override with !!kvm_x86_ops.gmem_make_shared is not too bad either:
+ doesn't leak anything, since the function being called is kvm_arch_gmem_make_shared and the accompanying function is kvm_arch_has_gmem_make_shared. Or maybe just a little, since all the other ops don't have the accompanying _has_ function
+ it's a kernel-internal thing
+ not too many lines of code, not too complex

It also provides a good excuse to kill off the #idfefs in guest_memfd.c. Compile tested only, but I'm thinking this?

From: Sean Christopherson <[email protected]>
Date: Mon, 17 Aug 2026 12:31:50 -0700
Subject: [PATCH] KVM: guest_memfd: Optimize away conversion overheads via
dead-code elimination

Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleed guest_memfd details into arch code and would be far worse than adding yet another kvm_arch_has... hook.

Opportunistically provide the kvm_arch_gmem_make_private() declaration, and rely on dead-code elimination to eliminate the call to non-existent code when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.

---
 arch/x86/include/asm/kvm_host.h | 3 +++
 include/linux/kvm_host.h        | 3 ++-
 virt/kvm/guest_memfd.c          | 5 ++---
 3 files changed, 7 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
index 283847619ff8..5d5a7723abb6 100644
@@ -1854,6 +1854,9 @@ enum kvm_intr_type {
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+#define kvm_arch_has_gmem_convert() (!!kvm_x86_ops.gmem_make_private)
+#endif
 
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
 
include/linux/kvm_host.h +2 -1
index 03bfc92864b6..e824ba59c60c 100644
@@ -2599,9 +2599,10 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			       kvm_pfn_t nr_pages);
+#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+#define kvm_arch_has_gmem_convert() false
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +2 -3
index b596486d184c..39d94938b5f6 100644
@@ -773,11 +773,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		folio_mark_uptodate(folio);
 	}
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
-	if (kvm_gmem_is_private_mem(file_inode(file), index))
+	if (kvm_arch_has_gmem_convert() &&
+	    kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
 					       (kvm_pfn_t)1 << *max_order);
-#endif
 
 	folio_unlock(folio);
 
Quoted 2 lines

base-commit: 1b731e5ded480bd1e5546aed35584238661ce72e --

I'll inserted this as the first patch for v11.

Then, the later patch "Call arch make_shared callback for to-shared conversion" is now:

	if (!to_private && kvm_arch_has_gmem_convert())
		kvm_gmem_make_shared(inode, start, end);

kvm_gmem_make_shared()'s definition is still guarded by #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT, and there's still a stub for kvm_gmem_make_shared() since kvm_gmem_make_shared() is defined and used in the same file and the optimizing out only happens later.

#85 lore raw

PATCH v10 13/41 KVM: guest_memfd: Return early if range already has requested attributes

In reply to Ackerley Tng

Provide function to check that a range has given attributes.

Optimize setting memory attributes by returning early if all pages in the requested range already has the requested attributes.

---
 virt/kvm/guest_memfd.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
virt/kvm/guest_memfd.c +22 -1
index c9874bd85c5d8..896699afcad9d 100644
@@ -95,6 +95,24 @@ static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)
 	return !kvm_gmem_is_private_mem(inode, index);
 }
 
+static bool kvm_gmem_range_has_attributes(struct inode *inode,
+					  pgoff_t index, size_t nr_pages,
+					  u64 attributes)
+{
+	struct maple_tree *mt = &GMEM_I(inode)->attributes;
+	pgoff_t end = index + nr_pages - 1;
+	void *entry;
+
+	lockdep_assert(mt_lock_is_held(mt));
+
+	mt_for_each(mt, entry, index, end) {
+		if (kvm_gmem_get_attributes(inode, entry) != attributes)
+			return false;
+	}
+
+	return true;
+}
+
 /*
  * Returns a locked folio on success.  The caller is responsible for
  * setting the up-to-date flag before the memory is mapped into the guest.
@@ -605,12 +623,15 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	pgoff_t end = start + nr_pages;
 	struct maple_tree *mt;
 	struct ma_state mas;
-	int r;
+	int r = 0;
 
 	mt = &gi->attributes;
 
 	filemap_invalidate_lock(mapping);
 
+	if (kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs))
+		goto out;
+
 	mas_init(&mas, mt, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
 	if (r) {
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 13/41
#86 lore raw

Re: [PATCH v10 13/41] KVM: guest_memfd: Return early if range already has requested attributes

Quoted 6 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Provide function to check that a range has given attributes.

Optimize setting memory attributes by returning early if all pages in the requested range already has the requested attributes.

Nit: has -> have

Quoted 4 lines
Reviewed-by on v10 13/41
#87 lore raw

Re: [PATCH v10 13/41] KVM: guest_memfd: Return early if range already has requested attributes

In reply to Ackerley Tng
Quoted 10 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Provide function to check that a range has given attributes.

Optimize setting memory attributes by returning early if all pages in the requested range already has the requested attributes.

Quoted 51 lines
---
  virt/kvm/guest_memfd.c | 23 ++++++++++++++++++++++-
  1 file changed, 22 insertions(+), 1 deletion(-)
virt/kvm/guest_memfd.c +22 -1
index c9874bd85c5d8..896699afcad9d 100644
@@ -95,6 +95,24 @@ static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)
  	return !kvm_gmem_is_private_mem(inode, index);
  }
 
+static bool kvm_gmem_range_has_attributes(struct inode *inode,
+					  pgoff_t index, size_t nr_pages,
+					  u64 attributes)
+{
+	struct maple_tree *mt = &GMEM_I(inode)->attributes;
+	pgoff_t end = index + nr_pages - 1;
+	void *entry;
+
+	lockdep_assert(mt_lock_is_held(mt));
+
+	mt_for_each(mt, entry, index, end) {
+		if (kvm_gmem_get_attributes(inode, entry) != attributes)
+			return false;
+	}
+
+	return true;
+}
+
  /*
   * Returns a locked folio on success.  The caller is responsible for
   * setting the up-to-date flag before the memory is mapped into the guest.
@@ -605,12 +623,15 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
  	pgoff_t end = start + nr_pages;
  	struct maple_tree *mt;
  	struct ma_state mas;
-	int r;
+	int r = 0;
 
  	mt = &gi->attributes;
 
  	filemap_invalidate_lock(mapping);
 
+	if (kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs))
+		goto out;
+
  	mas_init(&mas, mt, start);
  	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
  	if (r) {
#88 lore raw

PATCH v10 14/41 mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()

In reply to Ackerley Tng

From: "David Hildenbrand (Arm)" <[email protected]>

KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references).

So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit.

Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required.

Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well.

---
 include/linux/swap.h | 11 ++++++++++-
 mm/gup.c             | 15 ++-------------
 mm/swap.c            | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 14 deletions(-)
include/linux/swap.h +10 -1
index 8f0f68e245baa..e625d64c9e13d 100644
@@ -317,7 +317,7 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
 
-static inline bool folio_may_be_lru_cached(struct folio *folio)
+static inline bool folio_may_be_lru_cached(const struct folio *folio)
 {
 	/*
 	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
@@ -344,6 +344,15 @@ extern void lru_add_drain(void);
 extern void lru_add_drain_cpu(int cpu);
 extern void lru_add_drain_cpu_zone(struct zone *zone);
 extern void lru_add_drain_all(void);
+
+enum lru_cache_drained {
+	LRU_CACHE_NOT_DRAINED,
+	LRU_CACHE_DRAINED,
+	LRU_CACHE_DRAINED_ALL,
+};
+void lru_cache_drain_for_folio(const struct folio *folio,
+		unsigned int extra_refs, enum lru_cache_drained *drained);
+
 void folio_deactivate(struct folio *folio);
 void folio_mark_lazyfree(struct folio *folio);
 extern void swap_setup(void);
mm/gup.c +2 -13
index bde05664fe9a9..1bd8915517196 100644
@@ -2266,9 +2266,9 @@ static unsigned long collect_longterm_unpinnable_folios(
 		struct list_head *movable_folio_list,
 		struct pages_or_folios *pofs)
 {
+	enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
 	unsigned long collected = 0;
 	struct folio *folio;
-	int drained = 0;
 	long i = 0;
 
 	for (folio = pofs_get_folio(pofs, i); folio;
@@ -2293,18 +2293,7 @@ static unsigned long collect_longterm_unpinnable_folios(
 		 * but also to remove any other folio references from LRU
 		 * caches.
 		 */
-		if (drained == 0 && folio_may_be_lru_cached(folio) &&
-				folio_ref_count(folio) !=
-				folio_expected_ref_count(folio) + pin_refs) {
-			lru_add_drain();
-			drained = 1;
-		}
-		if (drained == 1 && folio_may_be_lru_cached(folio) &&
-				folio_ref_count(folio) !=
-				folio_expected_ref_count(folio) + pin_refs) {
-			lru_add_drain_all();
-			drained = 2;
-		}
+		lru_cache_drain_for_folio(folio, pin_refs, &drained);
 
 		if (!folio_isolate_lru(folio))
 			continue;
mm/swap.c +46 -0

52 lines not on this page · raw message

Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#89 lore raw

Re: [PATCH v10 14/41] mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()

Quoted 21 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: "David Hildenbrand (Arm)" <[email protected]>

KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references).

So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit.

Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required.

Quoted 6 lines

Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well.

---

If all goes well, this patch should go into 7.3-rc1. So once you rebase, you'd get it for free :)

Signature · 3 lines
Cheers,

David
Reviewed-by on v10 14/41
#90 lore raw

Re: [PATCH v10 14/41] mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()

In reply to Ackerley Tng
Quoted 22 lines · On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay…

From: "David Hildenbrand (Arm)" <[email protected]>

KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references).

So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit.

Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required.

Quoted 5 lines

Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well.

Cheers,
/fuad

Quoted 132 lines
---
 include/linux/swap.h | 11 ++++++++++-
 mm/gup.c             | 15 ++-------------
 mm/swap.c            | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 14 deletions(-)
include/linux/swap.h +10 -1
index 8f0f68e245baa..e625d64c9e13d 100644
@@ -317,7 +317,7 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
 
-static inline bool folio_may_be_lru_cached(struct folio *folio)
+static inline bool folio_may_be_lru_cached(const struct folio *folio)
 {
        /*
         * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
@@ -344,6 +344,15 @@ extern void lru_add_drain(void);
 extern void lru_add_drain_cpu(int cpu);
 extern void lru_add_drain_cpu_zone(struct zone *zone);
 extern void lru_add_drain_all(void);
+
+enum lru_cache_drained {
+       LRU_CACHE_NOT_DRAINED,
+       LRU_CACHE_DRAINED,
+       LRU_CACHE_DRAINED_ALL,
+};
+void lru_cache_drain_for_folio(const struct folio *folio,
+               unsigned int extra_refs, enum lru_cache_drained *drained);
+
 void folio_deactivate(struct folio *folio);
 void folio_mark_lazyfree(struct folio *folio);
 extern void swap_setup(void);
mm/gup.c +2 -13
index bde05664fe9a9..1bd8915517196 100644
@@ -2266,9 +2266,9 @@ static unsigned long collect_longterm_unpinnable_folios(
                struct list_head *movable_folio_list,
                struct pages_or_folios *pofs)
 {
+       enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
        unsigned long collected = 0;
        struct folio *folio;
-       int drained = 0;
        long i = 0;
 
        for (folio = pofs_get_folio(pofs, i); folio;
@@ -2293,18 +2293,7 @@ static unsigned long collect_longterm_unpinnable_folios(
                 * but also to remove any other folio references from LRU
                 * caches.
                 */
-               if (drained == 0 && folio_may_be_lru_cached(folio) &&
-                               folio_ref_count(folio) !=
-                               folio_expected_ref_count(folio) + pin_refs) {
-                       lru_add_drain();
-                       drained = 1;
-               }
-               if (drained == 1 && folio_may_be_lru_cached(folio) &&
-                               folio_ref_count(folio) !=
-                               folio_expected_ref_count(folio) + pin_refs) {
-                       lru_add_drain_all();
-                       drained = 2;
-               }
+               lru_cache_drain_for_folio(folio, pin_refs, &drained);
 
                if (!folio_isolate_lru(folio))
                        continue;
mm/swap.c +46 -1
index 588f50d8f1a8c..8e965c8ce9aa9 100644
@@ -950,6 +950,52 @@ void lru_add_drain_all(void)
 }
 #endif /* CONFIG_SMP */
 
+/**
+ * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold
+ *                              folio references
+ * @folio: The folio.
+ * @extra_refs: Extra folio references held by the caller.
+ * @drained: Drain status for batch folio processing.
+ *
+ * Drain LRU caches if the caches might hold folio references. Start
+ * with a local LRU cache drain, to then drain LRU caches on all CPUs if
+ * local draining was insufficient.
+ *
+ * This function detects LRU cache references by comparing the folio refcount
+ * with the sum of the expected folio refcount + extra references held by the
+ * caller. Note that we cannot rely on PG_lru to reliably detect all LRU
+ * cache references, and there are rare scenarios (concurrent folio (un)mapping)
+ * where this function might miss detecting LRU cache references.
+ *
+ * If @drained is not NULL, the function will avoid re-draining LRU caches
+ * when processing multiple folios in a row. In that case, the variable
+ * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before
+ * the first invocation by the caller.
+ */
+void lru_cache_drain_for_folio(const struct folio *folio,
+               unsigned int extra_refs, enum lru_cache_drained *drained)
+{
+       if (!folio_may_be_lru_cached(folio))
+               return;
+
+       if (!drained || *drained == LRU_CACHE_NOT_DRAINED) {
+               if (folio_ref_count(folio) ==
+                   folio_expected_ref_count(folio) + extra_refs)
+                       return;
+               lru_add_drain();
+               if (drained)
+                       *drained = LRU_CACHE_DRAINED;
+       }
+       if (!drained || *drained == LRU_CACHE_DRAINED) {
+               if (folio_ref_count(folio) ==
+                   folio_expected_ref_count(folio) + extra_refs)
+                       return;
+               lru_add_drain_all();
+               if (drained)
+                       *drained = LRU_CACHE_DRAINED_ALL;
+       }
+}
+
 atomic_t lru_disable_count = ATOMIC_INIT(0);
 
 /*
 
--

2.55.0.654.g21b8a5bc05-goog

#91 lore raw

PATCH v10 15/41 KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

In reply to Ackerley Tng

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

This one known "usage" of the folio is handled by draining the lru_add fbatch. After draining, if the refcount is still elevated, then there's truly some other user of this page, and the page is not safe for conversion.

If the page may be dma pinned, DMA is obviously using it and hence not safe for conversions. If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it is also obviously not safe for conversion. Exit early to avoid unnecessary draining in these 2 cases.

Provide a drain status to only drain once ever while processing a batch of folios.

---
 mm/swap.c              |  2 ++
 virt/kvm/guest_memfd.c | 23 +++++++++++++++++++----
 2 files changed, 21 insertions(+), 4 deletions(-)
mm/swap.c +2 -0
index 8e965c8ce9aa9..9f511b97ab110 100644
@@ -37,6 +37,7 @@
 #include <linux/page_idle.h>
 #include <linux/local_lock.h>
 #include <linux/buffer_head.h>
+#include <linux/kvm_types.h>
 
 #include "internal.h"
 
@@ -995,6 +996,7 @@ void lru_cache_drain_for_folio(const struct folio *folio,
 			*drained = LRU_CACHE_DRAINED_ALL;
 	}
 }
+EXPORT_SYMBOL_FOR_KVM(lru_cache_drain_for_folio);
 
 atomic_t lru_disable_count = ATOMIC_INIT(0);
 
virt/kvm/guest_memfd.c +19 -4
index 896699afcad9d..030af0855f8b0 100644
@@ -8,6 +8,7 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include <linux/swap.h>
 
 #include "kvm_mm.h"
 #include "guest_memfd.h"
@@ -542,11 +543,26 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
 	return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
+static bool __folio_safe_for_conversion(struct folio *folio,
+					enum lru_cache_drained *drained)
+{
+	const int filemap_get_folios_refcount = 1;
+
+	if (folio_maybe_dma_pinned(folio) || folio_mapped(folio))
+		return false;
+
+	lru_cache_drain_for_folio(folio, filemap_get_folios_refcount,
+				  drained);
+
+	return folio_ref_count(folio) ==
+	       folio_nr_pages(folio) + filemap_get_folios_refcount;
+}
+
 static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
 					    size_t nr_pages, pgoff_t *err_index)
 {
+	enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
 	struct address_space *mapping = inode->i_mapping;
-	const int filemap_get_folios_refcount = 1;
 	pgoff_t last = start + nr_pages - 1;
 	struct folio_batch fbatch;
 	bool safe = true;
@@ -560,9 +576,8 @@ static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
 		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
 			struct folio *folio = fbatch.folios[i];
 
-			if (folio_ref_count(folio) !=
-			    folio_nr_pages(folio) + filemap_get_folios_refcount) {
-				safe = false;
+			safe = __folio_safe_for_conversion(folio, &drained);
+			if (!safe) {
 				*err_index = max(start, folio->index);
 				break;
 			}
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#92 lore raw

Re: [PATCH v10 15/41] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

Quoted 11 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

As discussed, what would happen if we mlocked gmem folios? In that case, they could also have been added to the mlock cache (which we treat like an LRU cache).

But that's just a comment regarding the text here.

[...]

Signature · 3 lines
Cheers,

David
#93 lore raw

Re: [PATCH v10 15/41] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

Quoted 13 lines · David Hildenbrand (Arm)
Quoted 11 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

Quoted 7 lines

As discussed, what would happen if we mlocked gmem folios? In that case, they could also have been added to the mlock cache (which we treat like an LRU cache).

But that's just a comment regarding the text here.

[...]

To make sure I'll get the wording right, let me know what you think of this (will replace the entire commit message for v11):

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

A folio may also have extra refcounts if it is on the mlock fbatch.

These two known "usages" of the folio are handled calling lru_cache_drain_for_folio, which drains both the lru_add and mlock fbatches. After draining, if the refcount is still elevated, then there's truly some other user of this page, and the page is not safe for conversion.

If the page may be dma pinned, DMA is obviously using it and hence not safe for conversions. If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it is also obviously not safe for conversion. Exit early to avoid unnecessary draining in these 2 cases.

Provide a drain status to only drain once ever while processing a batch of folios.

Quoted 4 lines

--
Cheers,

David

reviewer Reviewed-by on v10 15/41
#94 lore raw

Re: [PATCH v10 15/41] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

In reply to Ackerley Tng
Quoted 23 lines · On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay…

From: Ackerley Tng <[email protected]>

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

This one known "usage" of the folio is handled by draining the lru_add fbatch. After draining, if the refcount is still elevated, then there's truly some other user of this page, and the page is not safe for conversion.

If the page may be dma pinned, DMA is obviously using it and hence not safe for conversions. If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it is also obviously not safe for conversion. Exit early to avoid unnecessary draining in these 2 cases.

Quoted 6 lines

Provide a drain status to only drain once ever while processing a batch of folios.

Cheers,
/fuad

Quoted 82 lines
---
 mm/swap.c              |  2 ++
 virt/kvm/guest_memfd.c | 23 +++++++++++++++++++----
 2 files changed, 21 insertions(+), 4 deletions(-)
mm/swap.c +2 -0
index 8e965c8ce9aa9..9f511b97ab110 100644
@@ -37,6 +37,7 @@
 #include <linux/page_idle.h>
 #include <linux/local_lock.h>
 #include <linux/buffer_head.h>
+#include <linux/kvm_types.h>
 
 #include "internal.h"
 
@@ -995,6 +996,7 @@ void lru_cache_drain_for_folio(const struct folio *folio,
                        *drained = LRU_CACHE_DRAINED_ALL;
        }
 }
+EXPORT_SYMBOL_FOR_KVM(lru_cache_drain_for_folio);
 
 atomic_t lru_disable_count = ATOMIC_INIT(0);
 
virt/kvm/guest_memfd.c +19 -5
index 896699afcad9d..030af0855f8b0 100644
@@ -8,6 +8,7 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include <linux/swap.h>
 
 #include "kvm_mm.h"
 #include "guest_memfd.h"
@@ -542,11 +543,26 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes,
        return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL);
 }
 
+static bool __folio_safe_for_conversion(struct folio *folio,
+                                       enum lru_cache_drained *drained)
+{
+       const int filemap_get_folios_refcount = 1;
+
+       if (folio_maybe_dma_pinned(folio) || folio_mapped(folio))
+               return false;
+
+       lru_cache_drain_for_folio(folio, filemap_get_folios_refcount,
+                                 drained);
+
+       return folio_ref_count(folio) ==
+              folio_nr_pages(folio) + filemap_get_folios_refcount;
+}
+
 static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
                                            size_t nr_pages, pgoff_t *err_index)
 {
+       enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
        struct address_space *mapping = inode->i_mapping;
-       const int filemap_get_folios_refcount = 1;
        pgoff_t last = start + nr_pages - 1;
        struct folio_batch fbatch;
        bool safe = true;
@@ -560,9 +576,8 @@ static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
                for (i = 0; i < folio_batch_count(&fbatch); ++i) {
                        struct folio *folio = fbatch.folios[i];
 
-                       if (folio_ref_count(folio) !=
-                           folio_nr_pages(folio) + filemap_get_folios_refcount) {
-                               safe = false;
+                       safe = __folio_safe_for_conversion(folio, &drained);
+                       if (!safe) {
                                *err_index = max(start, folio->index);
                                break;
                        }
 
--

2.55.0.654.g21b8a5bc05-goog

Reviewed-by on v10 15/41
#95 lore raw

Re: [PATCH v10 15/41] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

In reply to Ackerley Tng
Quoted 22 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

A guest_memfd folio is safe for conversion if guest_memfd holds the last references on it. Any other references on the folio may indicate another user, and guest_memfd cannot convert it to private if there may be an existing host user.

A folio will have extra refcounts if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing.

This one known "usage" of the folio is handled by draining the lru_add fbatch. After draining, if the refcount is still elevated, then there's truly some other user of this page, and the page is not safe for conversion.

If the page may be dma pinned, DMA is obviously using it and hence not safe for conversions. If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it is also obviously not safe for conversion. Exit early to avoid unnecessary draining in these 2 cases.

Quoted 6 lines

Provide a drain status to only drain once ever while processing a batch of folios.

With the updated version of change log,

One nit below.

[...]

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 550

Quoted 5 lines
virt/kvm/guest_memfd.c +5 -0
+static bool __folio_safe_for_conversion(struct folio *folio,
+					enum lru_cache_drained *drained)
+{
+	const int filemap_get_folios_refcount = 1;
+

Nit:
After adding the helper, the use of filemap_get_folios_refcount seems less obvious. It introduces unnecessary tight coupling with the caller, though it has only one caller currently.

Is it better to pass the value as extra_refs, similar to what lru_cache_drain_for_folio() does?

Quoted 32 lines
virt/kvm/guest_memfd.c +13 -4
+	if (folio_maybe_dma_pinned(folio) || folio_mapped(folio))
+		return false;
+
+	lru_cache_drain_for_folio(folio, filemap_get_folios_refcount,
+				  drained);
+
+	return folio_ref_count(folio) ==
+	       folio_nr_pages(folio) + filemap_get_folios_refcount;
+}
+
 static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
 					    size_t nr_pages, pgoff_t *err_index)
 {
+	enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
 	struct address_space *mapping = inode->i_mapping;
-	const int filemap_get_folios_refcount = 1;
 	pgoff_t last = start + nr_pages - 1;
 	struct folio_batch fbatch;
 	bool safe = true;
@@ -560,9 +576,8 @@ static bool kvm_gmem_is_safe_for_conversion(struct inode *inode, pgoff_t start,
 		for (i = 0; i < folio_batch_count(&fbatch); ++i) {
 			struct folio *folio = fbatch.folios[i];
 
-			if (folio_ref_count(folio) !=
-			    folio_nr_pages(folio) + filemap_get_folios_refcount) {
-				safe = false;
+			safe = __folio_safe_for_conversion(folio, &drained);
+			if (!safe) {
 				*err_index = max(start, folio->index);
 				break;
 			}
#96 lore raw

Re: [PATCH v10 15/41] KVM: guest_memfd: Handle lru_add fbatch refcounts during conversion safety check

Quoted 14 lines · Binbin Wu

[...snip...]

virt/kvm/guest_memfd.c static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, · line 550

Quoted 5 lines
virt/kvm/guest_memfd.c +5 -0
+static bool __folio_safe_for_conversion(struct folio *folio,
+					enum lru_cache_drained *drained)
+{
+	const int filemap_get_folios_refcount = 1;
+

Nit:
After adding the helper, the use of filemap_get_folios_refcount seems less obvious. It introduces unnecessary tight coupling with the caller, though it has only one caller currently.

Quoted 4 lines

Is it better to pass the value as extra_refs, similar to what lru_cache_drain_for_folio() does?

David also suggested [1] something similar, please see
https://lore.kernel.org/all/[email protected]/,

Please add on to that thread if you disagree with the suggestions!

Quoted 13 lines
Quoted 13 lines
virt/kvm/guest_memfd.c +10 -0
+	if (folio_maybe_dma_pinned(folio) || folio_mapped(folio))
+		return false;
+
+	lru_cache_drain_for_folio(folio, filemap_get_folios_refcount,
+				  drained);
+
+	return folio_ref_count(folio) ==
+	       folio_nr_pages(folio) + filemap_get_folios_refcount;
+}
+
 
[...snip...]
#97 lore raw

PATCH v10 16/41 KVM: guest_memfd: Zero page while getting pfn

In reply to Ackerley Tng

Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().

With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.

Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.

guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.

Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.

This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().

Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.

---
 virt/kvm/guest_memfd.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +5 -7
index 030af0855f8b0..41950c8105b5d 100644
@@ -1079,6 +1079,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 		return ERR_PTR(-EHWPOISON);
 	}
 
+	if (!folio_test_uptodate(folio)) {
+		clear_highpage(folio_page(folio, 0));
+		folio_mark_uptodate(folio);
+	}
+
 	*pfn = folio_file_pfn(folio, index);
 	if (max_order)
 		*max_order = 0;
@@ -1108,11 +1113,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
-
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	if (kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
@@ -1159,8 +1159,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
-	if (!ret)
-		folio_mark_uptodate(folio);
 
 out_put_folio:
 	folio_put(folio);
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#98 lore raw

Re: [PATCH v10 16/41] KVM: guest_memfd: Zero page while getting pfn

Quoted 21 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().

With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.

Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.

guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.

Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.

Quoted 5 lines

This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().

Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.

The downside is that __kvm_gmem_populate() will now zero+write. I assume we don't care about possible performance impacts?

Worth spelling that out.

Quoted 38 lines
---
 virt/kvm/guest_memfd.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +5 -5
index 030af0855f8b0..41950c8105b5d 100644
@@ -1079,6 +1079,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 		return ERR_PTR(-EHWPOISON);
 	}
 
+	if (!folio_test_uptodate(folio)) {
+		clear_highpage(folio_page(folio, 0));
+		folio_mark_uptodate(folio);
+	}
+
 	*pfn = folio_file_pfn(folio, index);
 	if (max_order)
 		*max_order = 0;
@@ -1108,11 +1113,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
-
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	if (kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,

virt/kvm/guest_memfd.c static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, · line 1163

Quoted 5 lines
virt/kvm/guest_memfd.c +0 -2
@@ -1159,8 +1159,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
-	if (!ret)
-		folio_mark_uptodate(folio);

In case post-populate failed, do we want to re-zero the pages?

Signature · 3 lines
Cheers,

David
#99 lore raw

Re: [PATCH v10 16/41] KVM: guest_memfd: Zero page while getting pfn

Quoted 22 lines · David Hildenbrand (Arm)
Quoted 21 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().

With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.

Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.

guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.

Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.

Quoted 9 lines
Quoted 5 lines

This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().

Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.

The downside is that __kvm_gmem_populate() will now zero+write. I assume we don't care about possible performance impacts?

Would we rather zero in kvm_gmem_populate() only if !uptodate && gmem_in_place_conversion && src_addr == 0? That could work too.

Previously, without in-place conversion, populate never reads memory from guest_memfd so there was no danger of leaking uninitialized memory.

Quoted 42 lines

Worth spelling that out.

Quoted 40 lines
---
 virt/kvm/guest_memfd.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +5 -5
index 030af0855f8b0..41950c8105b5d 100644
@@ -1079,6 +1079,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 		return ERR_PTR(-EHWPOISON);
 	}
 
+	if (!folio_test_uptodate(folio)) {
+		clear_highpage(folio_page(folio, 0));
+		folio_mark_uptodate(folio);
+	}
+
 	*pfn = folio_file_pfn(folio, index);
 	if (max_order)
 		*max_order = 0;
@@ -1108,11 +1113,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
-
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	if (kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
@@ -1159,8 +1159,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
Quoted 6 lines

virt/kvm/guest_memfd.c static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, · line 1163

Quoted 3 lines
virt/kvm/guest_memfd.c +0 -2
 	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
-	if (!ret)
-		folio_mark_uptodate(folio);

In case post-populate failed, do we want to re-zero the pages?

I believe we can't re-zero the pages. When SNP fails to populate it could be because SNP didn't like the CPUIDs userspace set up, and after the error userspace is expected to check what SNP likes, then retry.

IIUC zeroing will destroy the message SNP wanted to leave for userspace.

Michael should be able to explain more here :)

Btw Michael may I also get your reviews on this series?

Quoted 4 lines

--
Cheers,

David

#100 lore raw

Re: [PATCH v10 16/41] KVM: guest_memfd: Zero page while getting pfn

Quoted 30 lines · Ackerley Tng
Quoted 28 lines · David Hildenbrand (Arm)
Quoted 26 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().

With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.

Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.

guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.

Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.

This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().

Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.

Quoted 6 lines
Quoted 3 lines

The downside is that __kvm_gmem_populate() will now zero+write. I assume we don't care about possible performance impacts?

Would we rather zero in kvm_gmem_populate() only if !uptodate && gmem_in_place_conversion && src_addr == 0? That could work too.

No. I'm 99% certain we discussed this (multiple times?), and the consensus was that any performance penalties due to redundant zeroing would pale in comparison to the cost of actually assigning the page to the VM.

Quoted 11 lines

Previously, without in-place conversion, populate never reads memory from guest_memfd so there was no danger of leaking uninitialized memory.

Quoted 8 lines

virt/kvm/guest_memfd.c static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, · line 1163

Quoted 6 lines
virt/kvm/guest_memfd.c +0 -2
@@ -1159,8 +1159,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
-	if (!ret)
-		folio_mark_uptodate(folio);

In case post-populate failed, do we want to re-zero the pages?

Quoted 8 lines

I believe we can't re-zero the pages. When SNP fails to populate it could be because SNP didn't like the CPUIDs userspace set up, and after the error userspace is expected to check what SNP likes, then retry.

IIUC zeroing will destroy the message SNP wanted to leave for userspace.

Michael should be able to explain more here :)

Not Michael, but the above is correct. If firmware rejects a CPUID page, then KVM copies back the expected CPUID values provided by firmware.

That said, now that we have have @may_writeback_src we _could_ re-zero the page, i.e. only zero pages for which @may_writeback_src is %false.

And _that_ said, I vote "no". KVM zeros the memory mostly to ensure userspace can't read stale data, e.g. someone else's data. I don't think we need to guarantee that a failed populate() (or rather, whatever ioctl called into it) will leave memory in any particular state. It would be easier to document that the page contents may be modified on failure.

#101 lore raw

Re: [PATCH v10 16/41] KVM: guest_memfd: Zero page while getting pfn

Quoted 9 lines
Quoted 9 lines

Previously, without in-place conversion, populate never reads memory from guest_memfd so there was no danger of leaking uninitialized memory.

Quoted 2 lines

In case post-populate failed, do we want to re-zero the pages?

I believe we can't re-zero the pages. When SNP fails to populate it could be because SNP didn't like the CPUIDs userspace set up, and after the error userspace is expected to check what SNP likes, then retry.

Ah, okay.

Quoted 8 lines
Quoted 4 lines

IIUC zeroing will destroy the message SNP wanted to leave for userspace.

Michael should be able to explain more here :)

Not Michael, but the above is correct. If firmware rejects a CPUID page, then KVM copies back the expected CPUID values provided by firmware.

Quoted 8 lines

That said, now that we have have @may_writeback_src we _could_ re-zero the page, i.e. only zero pages for which @may_writeback_src is %false.

And _that_ said, I vote "no". KVM zeros the memory mostly to ensure userspace can't read stale data, e.g. someone else's data. I don't think we need to guarantee that a failed populate() (or rather, whatever ioctl called into it) will leave memory in any particular state. It would be easier to document that the page contents may be modified on failure.

Let's do that then!

Signature · 3 lines
Cheers,

David
Reviewed-by on v10 16/41
#102 lore raw

Re: [PATCH v10 16/41] KVM: guest_memfd: Zero page while getting pfn

In reply to Ackerley Tng
Quoted 24 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate().

With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls.

Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes.

guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag.

Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing.

This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping().

Quoted 7 lines

Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call.

Quoted 42 lines
---
 virt/kvm/guest_memfd.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +5 -7
index 030af0855f8b0..41950c8105b5d 100644
@@ -1079,6 +1079,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 		return ERR_PTR(-EHWPOISON);
 	}
 
+	if (!folio_test_uptodate(folio)) {
+		clear_highpage(folio_page(folio, 0));
+		folio_mark_uptodate(folio);
+	}
+
 	*pfn = folio_file_pfn(folio, index);
 	if (max_order)
 		*max_order = 0;
@@ -1108,11 +1113,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
-
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	if (kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
@@ -1159,8 +1159,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
-	if (!ret)
-		folio_mark_uptodate(folio);
 
 out_put_folio:
 	folio_put(folio);
#103 lore raw

PATCH v10 17/41 KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE

In reply to Ackerley Tng

From: Michael Roth <[email protected]>

Make the source page for populating an SNP guest_memfd instance optional if in-place conversion/population is enabled. If KVM can convert the page in-place, then it's possible for guest memory to be initialized directly from userspace by mmap()'ing the guest_memfd and writing to it while the corresponding GPA ranges are in a 'shared' state, before converting them to the 'private' state expected by KVM_SEV_SNP_LAUNCH_UPDATE.

Update the handling/documentation for KVM_SEV_SNP_LAUNCH_UPDATE to allow for 'uaddr' to be set to NULL when in-place conversion is enabled, which SNP_LAUNCH_UPDATE will then use to determine when it should/shouldn't copy in data from a separate memory location. Continue to enforce non-NULL when PRIVATE is tracked per-VM, not per-guest_memfd.

---
 Documentation/virt/kvm/x86/amd-memory-encryption.rst | 14 ++++++++++----
 arch/x86/kvm/svm/sev.c                               | 11 ++++++-----
 virt/kvm/kvm_main.c                                  |  1 +
 3 files changed, 17 insertions(+), 9 deletions(-)
Documentation/virt/kvm/x86/amd-memory-encryption.rst +10 -4
index bd04a908a8dbd..5977fbe33b98c 100644
@@ -503,7 +503,8 @@ secrets.
 
 It is required that the GPA ranges initialized by this command have had the
 KVM_MEMORY_ATTRIBUTE_PRIVATE attribute set in advance. See the documentation
-for KVM_SET_MEMORY_ATTRIBUTES for more details on this aspect.
+for KVM_SET_MEMORY_ATTRIBUTES/KVM_SET_MEMORY_ATTRIBUTES2 for more details on
+this aspect.
 
 Upon success, this command is not guaranteed to have processed the entire
 range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of
@@ -511,9 +512,14 @@ range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of
 remaining range that has yet to be processed. The caller should continue
 calling this command until those fields indicate the entire range has been
 processed, e.g. ``len`` is 0, ``gfn_start`` is equal to the last GFN in the
-range plus 1, and ``uaddr`` is the last byte of the userspace-provided source
-buffer address plus 1. In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO,
-``uaddr`` will be ignored completely.
+range plus 1, and ``uaddr`` (if specified) is the last byte of the
+userspace-provided source buffer address plus 1.
+
+In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO, ``uaddr`` will be
+ignored completely. For all other page types, ``uaddr`` is optional if in-place
+conversion is enabled (i.e. when the data had been written directly to
+guest_memfd while the page was in the shared state) and is required if in-place
+conversion is disabled.
 
 Parameters (in): struct  kvm_sev_snp_launch_update
 
arch/x86/kvm/svm/sev.c +6 -5
index fcb41dfde4c02..f708fccf71307 100644
@@ -2395,7 +2395,7 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	 */
 	if (ret && !snp_page_reclaim(kvm, pfn) &&
 	    sev_populate_args->type == KVM_SEV_SNP_PAGE_TYPE_CPUID &&
-	    sev_populate_args->fw_error == SEV_RET_INVALID_PARAM) {
+	    sev_populate_args->fw_error == SEV_RET_INVALID_PARAM && src_page) {
 		void *src_vaddr = kmap_local_page(src_page);
 		void *dst_vaddr = kmap_local_pfn(pfn);
 
@@ -2428,8 +2428,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	if (copy_from_user(&params, u64_to_user_ptr(argp->data), sizeof(params)))
 		return -EFAULT;
 
-	pr_debug("%s: GFN start 0x%llx length 0x%llx type %d flags %d\n", __func__,
-		 params.gfn_start, params.len, params.type, params.flags);
+	pr_debug("%s: GFN start 0x%llx length 0x%llx type %d flags %d src %llx\n", __func__,
+		 params.gfn_start, params.len, params.type, params.flags, params.uaddr);
 
 	if (!params.len || !PAGE_ALIGNED(params.len) || params.flags ||
 	    (params.type != KVM_SEV_SNP_PAGE_TYPE_NORMAL &&
@@ -2441,7 +2441,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 
 	if (params.type == KVM_SEV_SNP_PAGE_TYPE_ZERO)
 		src = NULL;
-	else if (!params.uaddr || !PAGE_ALIGNED(params.uaddr))
+	else if ((!gmem_in_place_conversion && !params.uaddr) ||
+		 !PAGE_ALIGNED(params.uaddr))
 		return -EINVAL;
 	else
 		src = u64_to_user_ptr(params.uaddr);
@@ -2488,7 +2489,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 
 	params.gfn_start += count;
 	params.len -= count * PAGE_SIZE;
-	if (params.type != KVM_SEV_SNP_PAGE_TYPE_ZERO)
+	if (src && params.type != KVM_SEV_SNP_PAGE_TYPE_ZERO)
 		params.uaddr += count * PAGE_SIZE;
 
 	if (copy_to_user(u64_to_user_ptr(argp->data), &params, sizeof(params)))
virt/kvm/kvm_main.c +1 -0
index c4c09593ef8b9..c8faf1cb82167 100644
@@ -104,6 +104,7 @@ module_param(allow_unsafe_mappings, bool, 0444);
 
 #ifdef kvm_arch_has_private_mem
 bool __ro_after_init gmem_in_place_conversion = false;
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(gmem_in_place_conversion);
 #endif
 
 /*
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#104 lore raw

PATCH v10 18/41 KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION

In reply to Ackerley Tng

Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.

Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.

Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.

Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.

---
 Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++
 arch/x86/kvm/vmx/tdx.c                   | 8 +++++---
 2 files changed, 9 insertions(+), 3 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +4 -0
index 6a222e9d09541..d8d9409120e61 100644
@@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION
 Initialize @nr_pages TDX guest private memory starting from @gpa with userspace
 provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned.
 
+If guest_memfd in-place conversion is enabled, pass 0 for @source_addr
+to represent "no source page". A source page is required if in-place
+conversion is not enabled or not supported.
+
 Note, before calling this sub command, memory attribute of the range
 [gpa, gpa + nr_pages] needs to be private.  Userspace can use
 KVM_SET_MEMORY_ATTRIBUTES to set the attribute.
arch/x86/kvm/vmx/tdx.c +5 -3
index b272c20586a74..7654a9f5fa128 100644
@@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm))
 		return -EIO;
 
-	kvm_tdx->page_add_src = src_page;
+	kvm_tdx->page_add_src = src_page ?: pfn_to_page(pfn);
 	ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn);
 	kvm_tdx->page_add_src = NULL;
 
@@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 	if (copy_from_user(&region, u64_to_user_ptr(cmd->data), sizeof(region)))
 		return -EFAULT;
 
-	if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr ||
+	if (!PAGE_ALIGNED(region.source_addr) ||
+	    (!gmem_in_place_conversion && !region.source_addr) ||
 	    !PAGE_ALIGNED(region.gpa) || !region.nr_pages ||
 	    region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa ||
 	    !vt_is_tdx_private_gpa(kvm, region.gpa) ||
@@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 			break;
 		}
 
-		region.source_addr += PAGE_SIZE;
+		if (region.source_addr)
+			region.source_addr += PAGE_SIZE;
 		region.gpa += PAGE_SIZE;
 		region.nr_pages--;
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 18/41
#105 lore raw

Re: [PATCH v10 18/41] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION

Quoted 14 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.

Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.

Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.

Quoted 5 lines

Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.

Nit:
Is Sean's SoB intentional here?

Quoted 2 lines
Quoted 54 lines
---
 Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++
 arch/x86/kvm/vmx/tdx.c                   | 8 +++++---
 2 files changed, 9 insertions(+), 3 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +4 -0
index 6a222e9d09541..d8d9409120e61 100644
@@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION
 Initialize @nr_pages TDX guest private memory starting from @gpa with userspace
 provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned.
 
+If guest_memfd in-place conversion is enabled, pass 0 for @source_addr
+to represent "no source page". A source page is required if in-place
+conversion is not enabled or not supported.
+
 Note, before calling this sub command, memory attribute of the range
 [gpa, gpa + nr_pages] needs to be private.  Userspace can use
 KVM_SET_MEMORY_ATTRIBUTES to set the attribute.
arch/x86/kvm/vmx/tdx.c +5 -3
index b272c20586a74..7654a9f5fa128 100644
@@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm))
 		return -EIO;
 
-	kvm_tdx->page_add_src = src_page;
+	kvm_tdx->page_add_src = src_page ?: pfn_to_page(pfn);
 	ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn);
 	kvm_tdx->page_add_src = NULL;
 
@@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 	if (copy_from_user(&region, u64_to_user_ptr(cmd->data), sizeof(region)))
 		return -EFAULT;
 
-	if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr ||
+	if (!PAGE_ALIGNED(region.source_addr) ||
+	    (!gmem_in_place_conversion && !region.source_addr) ||
 	    !PAGE_ALIGNED(region.gpa) || !region.nr_pages ||
 	    region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa ||
 	    !vt_is_tdx_private_gpa(kvm, region.gpa) ||
@@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 			break;
 		}
 
-		region.source_addr += PAGE_SIZE;
+		if (region.source_addr)
+			region.source_addr += PAGE_SIZE;
 		region.gpa += PAGE_SIZE;
 		region.nr_pages--;
 
#106 lore raw

Re: [PATCH v10 18/41] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION

Quoted 15 lines · Binbin Wu
Quoted 14 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.

Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.

Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.

Quoted 8 lines
Quoted 5 lines

Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.

Nit:
Is Sean's SoB intentional here?

Kinda sorta? I handed commits to Ackerley after rebasing them locally at one point, which is how my SoB got here. I think it makes sense to just drop them from the patches where I'm not listed as the author or Co-developed-by. In the end, I expect to apply this series and so my SoB will get added backon anyways.

Tested-by Reviewed-by on v10 18/41
#107 lore raw

Re: [PATCH v10 18/41] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION

In reply to Ackerley Tng
Quoted 65 lines · On Fri, Aug 07, 2026 at 02:52:57PM -0700, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.

Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.

Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.

Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.

---
 Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++
 arch/x86/kvm/vmx/tdx.c                   | 8 +++++---
 2 files changed, 9 insertions(+), 3 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +4 -0
index 6a222e9d09541..d8d9409120e61 100644
@@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION
 Initialize @nr_pages TDX guest private memory starting from @gpa with userspace
 provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned.
 
+If guest_memfd in-place conversion is enabled, pass 0 for @source_addr
+to represent "no source page". A source page is required if in-place
+conversion is not enabled or not supported.
+
 Note, before calling this sub command, memory attribute of the range
 [gpa, gpa + nr_pages] needs to be private.  Userspace can use
 KVM_SET_MEMORY_ATTRIBUTES to set the attribute.
arch/x86/kvm/vmx/tdx.c +3 -2
index b272c20586a74..7654a9f5fa128 100644
@@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm))
 		return -EIO;
 
-	kvm_tdx->page_add_src = src_page;
+	kvm_tdx->page_add_src = src_page ?: pfn_to_page(pfn);
 	ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn);
 	kvm_tdx->page_add_src = NULL;
 
@@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 	if (copy_from_user(&region, u64_to_user_ptr(cmd->data), sizeof(region)))
 		return -EFAULT;
 
-	if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr ||
+	if (!PAGE_ALIGNED(region.source_addr) ||
+	    (!gmem_in_place_conversion && !region.source_addr) ||
 	    !PAGE_ALIGNED(region.gpa) || !region.nr_pages ||
 	    region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa ||
 	    !vt_is_tdx_private_gpa(kvm, region.gpa) ||

arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3273

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +2 -1
@@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
 			break;
 		}
 
-		region.source_addr += PAGE_SIZE;
+		if (region.source_addr)
+			region.source_addr += PAGE_SIZE;
 		region.gpa += PAGE_SIZE;
 		region.nr_pages--;

Tested the following scenarios:
1.gmem_in_place_conversion == true,
(1) region.source_addr == NULL.

      ==> TD launched successfully.

(2) region.source_addr != NULL,
region.source_addr is not from mmap() of the guest_memfd (double backends) ==> TD launched successfully.

(3) region.source_addr != NULL,
region.source_addr is from mmap() of the guest_memfd (single backend), GPA is private.
==> Failed (-EFAULT).

(4) region.source_addr != NULL,
region.source_addr is from mmap() of the guest_memfd (single backend), GPA is shared.
==> Failed (-EINVAL).

2. gmem_in_place_conversion == false,
(1) region.source_addr == NULL.

      ==> Failed (-EINVAL).

(2) region.source_addr is not from mmap() of the guest_memfd.

      ==> TD launched successfully.

So,

Reviewed-by on v10 18/41
#108 lore raw

Re: [PATCH v10 18/41] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION

In reply to Ackerley Tng
Quoted 14 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN.

Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated.

Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd.

Quoted 7 lines

Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible.

Quoted 54 lines
---
  Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++
  arch/x86/kvm/vmx/tdx.c                   | 8 +++++---
  2 files changed, 9 insertions(+), 3 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +4 -0
index 6a222e9d09541..d8d9409120e61 100644
@@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION
  Initialize @nr_pages TDX guest private memory starting from @gpa with userspace
  provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned.
 
+If guest_memfd in-place conversion is enabled, pass 0 for @source_addr
+to represent "no source page". A source page is required if in-place
+conversion is not enabled or not supported.
+
  Note, before calling this sub command, memory attribute of the range
  [gpa, gpa + nr_pages] needs to be private.  Userspace can use
  KVM_SET_MEMORY_ATTRIBUTES to set the attribute.
arch/x86/kvm/vmx/tdx.c +5 -3
index b272c20586a74..7654a9f5fa128 100644
@@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
  	if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm))
  		return -EIO;
 
-	kvm_tdx->page_add_src = src_page;
+	kvm_tdx->page_add_src = src_page ?: pfn_to_page(pfn);
  	ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn);
  	kvm_tdx->page_add_src = NULL;
 
@@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
  	if (copy_from_user(&region, u64_to_user_ptr(cmd->data), sizeof(region)))
  		return -EFAULT;
 
-	if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr ||
+	if (!PAGE_ALIGNED(region.source_addr) ||
+	    (!gmem_in_place_conversion && !region.source_addr) ||
  	    !PAGE_ALIGNED(region.gpa) || !region.nr_pages ||
  	    region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa ||
  	    !vt_is_tdx_private_gpa(kvm, region.gpa) ||
@@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c
  			break;
  		}
 
-		region.source_addr += PAGE_SIZE;
+		if (region.source_addr)
+			region.source_addr += PAGE_SIZE;
  		region.gpa += PAGE_SIZE;
  		region.nr_pages--;
 
#109 lore raw

PATCH v10 19/41 KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Bury KVM_VM_MEMORY_ATTRIBUTES in x86 to discourage other architectures from adding support for per-VM memory attributes, because tracking private vs. shared memory on a per-VM basis is now deprecated in favor of tracking on a per-guest_memfd basis, and while RWX memory attributes are on the horizon, they too are expected to be x86-only.

This will also allow modifying KVM_VM_MEMORY_ATTRIBUTES to be user-selectable (in x86) without creating weirdness in KVM's Kconfigs. Now that guest_memfd supports in-place conversions, it's entirely possible to run x86 CoCo VMs without support for KVM_VM_MEMORY_ATTRIBUTES.

Leave the code itself in common KVM so that it's trivial to undo this change if new per-VM attributes do come along.

---
 arch/x86/kvm/Kconfig | 3 +++
 virt/kvm/Kconfig     | 3 ---
 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/Kconfig +3 -0
index 2f8e1b5a8a72c..abb108886733a 100644
@@ -80,6 +80,9 @@ config KVM_WERROR
 
 	  If in doubt, say "N".
 
+config KVM_VM_MEMORY_ATTRIBUTES
+	bool
+
 config KVM_SW_PROTECTED_VM
 	bool "Enable support for KVM software-protected VMs"
 	depends on EXPERT
virt/kvm/Kconfig +0 -3
index df575449d97b5..a0678ef8ee3f5 100644
@@ -100,9 +100,6 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG
 config KVM_MMU_LOCKLESS_AGING
        bool
 
-config KVM_VM_MEMORY_ATTRIBUTES
-       bool
-
 config KVM_GUEST_MEMFD
        select XARRAY_MULTI
        bool
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
maintainer Reviewed-by on v10 19/41
#110 lore raw

Re: [PATCH v10 19/41] KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86

Quoted 16 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Bury KVM_VM_MEMORY_ATTRIBUTES in x86 to discourage other architectures from adding support for per-VM memory attributes, because tracking private vs. shared memory on a per-VM basis is now deprecated in favor of tracking on a per-guest_memfd basis, and while RWX memory attributes are on the horizon, they too are expected to be x86-only.

This will also allow modifying KVM_VM_MEMORY_ATTRIBUTES to be user-selectable (in x86) without creating weirdness in KVM's Kconfigs. Now that guest_memfd supports in-place conversions, it's entirely possible to run x86 CoCo VMs without support for KVM_VM_MEMORY_ATTRIBUTES.

Leave the code itself in common KVM so that it's trivial to undo this change if new per-VM attributes do come along.

Quoted 7 lines

---

Signature · 3 lines
Cheers,

David
#111 lore raw

PATCH v10 20/41 KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

In reply to Ackerley Tng

Make gmem_in_place_conversion a module parameter so that userspace can configure enable or disable the use of VM-level memory attributes. The module parameter is only available if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled.

To avoid inconsistencies in the way memory attributes are tracked in KVM and guest_memfd, the vm_memory_attributes module_param is made read-only (0444).

Since selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES disables in-place conversion, actually make CONFIG_KVM_VM_MEMORY_ATTRIBUTES selectable. Make the config only selectable for (CoCo) VM types that might use vm_memory_attributes.

Since memory attributes are trackable in guest_memfd, the concept of having private memory is no longer dependent on CONFIG_KVM_VM_MEMORY_ATTRIBUTES. Define kvm_arch_has_private_mem() based on platform config, so that having private memory is dependent on (CoCo) VM type.

[Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled]
---
 arch/x86/include/asm/kvm_host.h |  4 +++-
 arch/x86/kvm/Kconfig            | 14 ++++++++++----
 virt/kvm/kvm_main.c             |  5 ++++-
 3 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -1
index 497da6aaf2942..bb2c1ade62cf9 100644
@@ -1851,7 +1851,9 @@ enum kvm_intr_type {
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest && \
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
-#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+#if defined(CONFIG_KVM_SW_PROTECTED_VM) ||	\
+    defined(CONFIG_KVM_INTEL_TDX) ||		\
+    defined(CONFIG_KVM_AMD_SEV)
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
 
arch/x86/kvm/Kconfig +10 -4
index abb108886733a..2c3c22aeafa54 100644
@@ -81,13 +81,21 @@ config KVM_WERROR
 	  If in doubt, say "N".
 
 config KVM_VM_MEMORY_ATTRIBUTES
-	bool
+	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
+	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
+	help
+	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
+	  memory attributes.  Using per-VM attributes is deprecated in favor of
+	  tracking PRIVATE state in guest_memfd.  Select this if you need to run
+	  CoCo VMs using a VMM that doesn't support guest_memfd memory
+	  attributes.
+
+	  If unsure, say N.
 
 config KVM_SW_PROTECTED_VM
 	bool "Enable support for KVM software-protected VMs"
 	depends on EXPERT
 	depends on KVM_X86 && X86_64
-	select KVM_VM_MEMORY_ATTRIBUTES
 	help
 	  Enable support for KVM software-protected VMs.  Currently, software-
 	  protected VMs are purely a development and testing vehicle for
@@ -138,7 +146,6 @@ config KVM_INTEL_TDX
 	bool "Intel Trust Domain Extensions (TDX) support"
 	default y
 	depends on INTEL_TDX_HOST
-	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
 	  Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -162,7 +169,6 @@ config KVM_AMD_SEV
 	depends on KVM_AMD && X86_64
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
-	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
virt/kvm/kvm_main.c +4 -1
index c8faf1cb82167..85dfc3f5af805 100644
@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;
 module_param(allow_unsafe_mappings, bool, 0444);
 
 #ifdef kvm_arch_has_private_mem
-bool __ro_after_init gmem_in_place_conversion = false;
+bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES);
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+module_param(gmem_in_place_conversion, bool, 0444);
+#endif
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(gmem_in_place_conversion);
 #endif
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#112 lore raw

Re: [PATCH v10 20/41] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

Quoted 59 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Make gmem_in_place_conversion a module parameter so that userspace can configure enable or disable the use of VM-level memory attributes. The module parameter is only available if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled.

To avoid inconsistencies in the way memory attributes are tracked in KVM and guest_memfd, the vm_memory_attributes module_param is made read-only (0444).

Since selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES disables in-place conversion, actually make CONFIG_KVM_VM_MEMORY_ATTRIBUTES selectable. Make the config only selectable for (CoCo) VM types that might use vm_memory_attributes.

Since memory attributes are trackable in guest_memfd, the concept of having private memory is no longer dependent on CONFIG_KVM_VM_MEMORY_ATTRIBUTES. Define kvm_arch_has_private_mem() based on platform config, so that having private memory is dependent on (CoCo) VM type.

[Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled]
---
 arch/x86/include/asm/kvm_host.h |  4 +++-
 arch/x86/kvm/Kconfig            | 14 ++++++++++----
 virt/kvm/kvm_main.c             |  5 ++++-
 3 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -1
index 497da6aaf2942..bb2c1ade62cf9 100644
@@ -1851,7 +1851,9 @@ enum kvm_intr_type {
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest && \
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
-#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+#if defined(CONFIG_KVM_SW_PROTECTED_VM) ||	\
+    defined(CONFIG_KVM_INTEL_TDX) ||		\
+    defined(CONFIG_KVM_AMD_SEV)
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
 
arch/x86/kvm/Kconfig +0 -1
index abb108886733a..2c3c22aeafa54 100644
@@ -81,13 +81,21 @@ config KVM_WERROR
 	  If in doubt, say "N".
 
 config KVM_VM_MEMORY_ATTRIBUTES
-	bool

arch/x86/kvm/Kconfig config KVM_WERROR · line 93

Quoted 10 lines
arch/x86/kvm/Kconfig +10 -0
+	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
+	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
+	help
+	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
+	  memory attributes.  Using per-VM attributes is deprecated in favor of
+	  tracking PRIVATE state in guest_memfd.  Select this if you need to run
+	  CoCo VMs using a VMM that doesn't support guest_memfd memory
+	  attributes.
+
+	  If unsure, say N.

Hm, does that imply that a distribution that must support both, cannot ship a single kernel?

Signature · 3 lines
Cheers,

David
#113 lore raw

Re: [PATCH v10 20/41] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

Quoted 13 lines · David Hildenbrand (Arm)
Quoted 12 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

[...snip...]

arch/x86/kvm/Kconfig +0 -0
index abb108886733a..2c3c22aeafa54 100644
@@ -81,13 +81,21 @@ config KVM_WERROR
 	  If in doubt, say "N".
 
Quoted 16 lines

arch/x86/kvm/Kconfig config KVM_WERROR · line 93

Quoted 12 lines
arch/x86/kvm/Kconfig +10 -1
 config KVM_VM_MEMORY_ATTRIBUTES
-	bool
+	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
+	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
+	help
+	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
+	  memory attributes.  Using per-VM attributes is deprecated in favor of
+	  tracking PRIVATE state in guest_memfd.  Select this if you need to run
+	  CoCo VMs using a VMM that doesn't support guest_memfd memory
+	  attributes.
+
+	  If unsure, say N.

Hm, does that imply that a distribution that must support both, cannot ship a single kernel?

Did you mean this as a documentation comment? Should the above read:

help
Enable support for tracking PRIVATE vs. SHARED memory using per-VM memory attributes. Using per-VM attributes is deprecated in favor of tracking PRIVATE state in guest_memfd. Select this if you need to run CoCo VMs using a VMM that doesn't support guest_memfd memory

-	  attributes.
+	  attributes. Select this to defer selection of where PRIVATE vs. SHARED
+	  attributes are tracked to KVM module load time, where
+	  gmem_in_place_conversion can be specified as a module parameter.

If unsure, say N.

I think we should do _without_ the above clarification since using VM memory attributes for shared/private status is intended to be deprecated.

Is it fair to say distributions should not support both, and if they _must_ support both, they need to read code to figure this out?

Quoted 4 lines

--
Cheers,

David

#114 lore raw

Re: [PATCH v10 20/41] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

Quoted 32 lines · Ackerley Tng
Quoted 13 lines · David Hildenbrand (Arm)
Quoted 12 lines · On 8/7/26 23:52, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

[...snip...]

arch/x86/kvm/Kconfig +0 -0
index abb108886733a..2c3c22aeafa54 100644
@@ -81,13 +81,21 @@ config KVM_WERROR
 	  If in doubt, say "N".
 
Quoted 16 lines

arch/x86/kvm/Kconfig config KVM_WERROR · line 93

Quoted 12 lines
arch/x86/kvm/Kconfig +10 -1
 config KVM_VM_MEMORY_ATTRIBUTES
-	bool
+	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
+	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
+	help
+	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
+	  memory attributes.  Using per-VM attributes is deprecated in favor of
+	  tracking PRIVATE state in guest_memfd.  Select this if you need to run
+	  CoCo VMs using a VMM that doesn't support guest_memfd memory
+	  attributes.
+
+	  If unsure, say N.

Hm, does that imply that a distribution that must support both, cannot ship a single kernel?

Quoted 1 line

Did you mean this as a documentation comment? Should the above read:

No, I was trying to understand the impact :)

IIUC, a distribution that would have to support both would simply enable CONFIG_KVM_VM_MEMORY_ATTRIBUTES, and the module parameter would have to be used to select the mode of operation.

I guess that's good enough!

Quoted 11 lines

help
Enable support for tracking PRIVATE vs. SHARED memory using per-VM memory attributes. Using per-VM attributes is deprecated in favor of tracking PRIVATE state in guest_memfd. Select this if you need to run CoCo VMs using a VMM that doesn't support guest_memfd memory

(quoted diff) +3 -1
-	  attributes.
+	  attributes. Select this to defer selection of where PRIVATE vs. SHARED
+	  attributes are tracked to KVM module load time, where
+	  gmem_in_place_conversion can be specified as a module parameter.
Quoted 8 lines

If unsure, say N.

I think we should do _without_ the above clarification since using VM memory attributes for shared/private status is intended to be deprecated.

Is it fair to say distributions should not support both, and if they _must_ support both, they need to read code to figure this out?

Yeah, I don't think we need a doc update.

Signature · 3 lines
Cheers,

David
#115 lore raw

Re: [PATCH v10 20/41] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

In reply to Ackerley Tng
Quoted 9 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Make gmem_in_place_conversion a module parameter so that userspace can configure enable or disable the use of VM-level memory attributes. The module parameter is only available if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled.

To avoid inconsistencies in the way memory attributes are tracked in KVM and guest_memfd, the vm_memory_attributes module_param is made

The description is stale, since there is no module_param called vm_memory_attributes?

Quoted 4 lines

read-only (0444).

Since selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES disables in-place conversion,

"selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES" doesn't necessarily disable in-place conversion, it also depends on the setting of gmem_in_place_conversion.
To be accurate, maybe add "by default"?

actually make CONFIG_KVM_VM_MEMORY_ATTRIBUTES selectable. Make

Quoted 98 lines

the config only selectable for (CoCo) VM types that might use vm_memory_attributes.

Since memory attributes are trackable in guest_memfd, the concept of having private memory is no longer dependent on CONFIG_KVM_VM_MEMORY_ATTRIBUTES. Define kvm_arch_has_private_mem() based on platform config, so that having private memory is dependent on (CoCo) VM type.

[Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled]
---
 arch/x86/include/asm/kvm_host.h |  4 +++-
 arch/x86/kvm/Kconfig            | 14 ++++++++++----
 virt/kvm/kvm_main.c             |  5 ++++-
 3 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -1
index 497da6aaf2942..bb2c1ade62cf9 100644
@@ -1851,7 +1851,9 @@ enum kvm_intr_type {
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest && \
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
-#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+#if defined(CONFIG_KVM_SW_PROTECTED_VM) ||	\
+    defined(CONFIG_KVM_INTEL_TDX) ||		\
+    defined(CONFIG_KVM_AMD_SEV)
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #endif
 
arch/x86/kvm/Kconfig +10 -4
index abb108886733a..2c3c22aeafa54 100644
@@ -81,13 +81,21 @@ config KVM_WERROR
 	  If in doubt, say "N".
 
 config KVM_VM_MEMORY_ATTRIBUTES
-	bool
+	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
+	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
+	help
+	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
+	  memory attributes.  Using per-VM attributes is deprecated in favor of
+	  tracking PRIVATE state in guest_memfd.  Select this if you need to run
+	  CoCo VMs using a VMM that doesn't support guest_memfd memory
+	  attributes.
+
+	  If unsure, say N.
 
 config KVM_SW_PROTECTED_VM
 	bool "Enable support for KVM software-protected VMs"
 	depends on EXPERT
 	depends on KVM_X86 && X86_64
-	select KVM_VM_MEMORY_ATTRIBUTES
 	help
 	  Enable support for KVM software-protected VMs.  Currently, software-
 	  protected VMs are purely a development and testing vehicle for
@@ -138,7 +146,6 @@ config KVM_INTEL_TDX
 	bool "Intel Trust Domain Extensions (TDX) support"
 	default y
 	depends on INTEL_TDX_HOST
-	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
 	  Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -162,7 +169,6 @@ config KVM_AMD_SEV
 	depends on KVM_AMD && X86_64
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
-	select KVM_VM_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
virt/kvm/kvm_main.c +4 -1
index c8faf1cb82167..85dfc3f5af805 100644
@@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings;
 module_param(allow_unsafe_mappings, bool, 0444);
 
 #ifdef kvm_arch_has_private_mem
-bool __ro_after_init gmem_in_place_conversion = false;
+bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES);
+#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+module_param(gmem_in_place_conversion, bool, 0444);
+#endif
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(gmem_in_place_conversion);
 #endif
 
#116 lore raw

Re: [PATCH v10 20/41] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes

Quoted 16 lines · Binbin Wu
Quoted 9 lines · On 8/8/2026 5:52 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng <[email protected]>

Make gmem_in_place_conversion a module parameter so that userspace can configure enable or disable the use of VM-level memory attributes. The module parameter is only available if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled.

To avoid inconsistencies in the way memory attributes are tracked in KVM and guest_memfd, the vm_memory_attributes module_param is made

The description is stale, since there is no module_param called vm_memory_attributes?

Quoted 2 lines

read-only (0444).

Quoted 7 lines
Quoted 2 lines

Since selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES disables in-place conversion,

"selecting CONFIG_KVM_VM_MEMORY_ATTRIBUTES" doesn't necessarily disable in-place conversion, it also depends on the setting of gmem_in_place_conversion.
To be accurate, maybe add "by default"?

+1.

Ackerley, please write changelogs in imperative mood, i.e. state things like this as command, not as a passive description of what the code now does. And I would omit the blurb on changing the kvm_arch_has_private_mem() definition, for me that falls into the category of giving a play-by-play explanation of the code change. I.e. Let the diff speak for itself.

E.g.

Allow the user to disable KVM_VM_MEMORY_ATTRIBUTES even when KVM supports PRIVATE and SHARED attributes, and expose gmem_in_place_conversion as a module parameter when per-VM attributes are supported. I.e. let userspace enable in-place PRIVATE<=>SHARED conversion of guest_memfd pages.

Provide both a Kconfig option and a (conditional) module param so that deployments that use a custom kernel can fully disable per-VM tracking, while not forcing distros to ship two separate kernels in order to provide backwards compatibility for downstream users.

Don't allow running VMs with mixed tracking for a given instance of KVM, i.e. disallow toggling the module param after KVM is loaded, as the extra complexity needed to handle per-VM behavior far outweighs any potential benefit. E.g. neither TDX nor SNP supports live migration, so in effect the requirement is that existing deployments that want to support both the old and the new models would need to tell their VMM which flavor of tracking to use.

#117 lore raw

PATCH v10 21/41 KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Now that guest_memfd supports tracking private vs. shared within gmem itself, allow userspace to specify INIT_SHARED on a guest_memfd instance for x86 Confidential Computing (CoCo) VMs, so long as in-place conversion is enabled, i.e. when it's actually possible for a guest_memfd instance to contain shared memory.

---
 arch/x86/kvm/x86.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/kvm/x86.c +7 -6
index 39968610868fd..1ac60628b4c07 100644
@@ -10631,14 +10631,15 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
 }
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
-/*
- * KVM doesn't yet support initializing guest_memfd memory as shared for VMs
- * with private memory (the private vs. shared tracking needs to be moved into
- * guest_memfd).
- */
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 {
-	return !kvm_arch_has_private_mem(kvm);
+	/*
+	 * INIT_SHARED is supported if in-place conversion is enabled, or if
+	 * the VM doesn't support private memory.  If the VM has private memory
+	 * and in-place conversion is disabled, then guest_memfd can _only_ be
+	 * used for private memory.
+	 */
+	return gmem_in_place_conversion || !kvm_arch_has_private_mem(kvm);
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
Reviewed-by on v10 21/41
#118 lore raw

Re: [PATCH v10 21/41] KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs

Quoted 8 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Sean Christopherson <[email protected]>

Now that guest_memfd supports tracking private vs. shared within gmem itself, allow userspace to specify INIT_SHARED on a guest_memfd instance for x86 Confidential Computing (CoCo) VMs, so long as in-place conversion is enabled, i.e. when it's actually possible for a guest_memfd instance to contain shared memory.

Quoted 5 lines
#119 lore raw

PATCH v10 22/41 KVM: selftests: Create gmem fd before "regular" fd when adding memslot

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

When adding a memslot associated a guest_memfd instance, create/dup the guest_memfd before creating the "normal" backing file. This will allow dup'ing the gmem fd as the normal fd when guest_memfd supports mmap(), i.e. to make guest_memfd the _only_ backing source for the memslot.

---
 tools/testing/selftests/kvm/lib/kvm_util.c | 45 +++++++++++++++---------------
 1 file changed, 23 insertions(+), 22 deletions(-)
tools/testing/selftests/kvm/lib/kvm_util.c +23 -22
index 9ddc047d5c275..3413d45e34faf 100644
@@ -1077,6 +1077,29 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 	if (alignment > 1)
 		region->mmap_size += alignment;
 
+	if (flags & KVM_MEM_GUEST_MEMFD) {
+		if (guest_memfd < 0) {
+			u32 guest_memfd_flags = 0;
+
+			TEST_ASSERT(!guest_memfd_offset,
+				    "Offset must be zero when creating new guest_memfd");
+			guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags);
+		} else {
+			/*
+			 * Install a unique fd for each memslot so that the fd
+			 * can be closed when the region is deleted without
+			 * needing to track if the fd is owned by the framework
+			 * or by the caller.
+			 */
+			guest_memfd = kvm_dup(guest_memfd);
+		}
+
+		region->region.guest_memfd = guest_memfd;
+		region->region.guest_memfd_offset = guest_memfd_offset;
+	} else {
+		region->region.guest_memfd = -1;
+	}
+
 	region->fd = -1;
 	if (backing_src_is_shared(src_type))
 		region->fd = kvm_memfd_alloc(region->mmap_size,
@@ -1106,28 +1129,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 
 	region->backing_src_type = src_type;
 
-	if (flags & KVM_MEM_GUEST_MEMFD) {
-		if (guest_memfd < 0) {
-			u32 guest_memfd_flags = 0;
-			TEST_ASSERT(!guest_memfd_offset,
-				    "Offset must be zero when creating new guest_memfd");
-			guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags);
-		} else {
-			/*
-			 * Install a unique fd for each memslot so that the fd
-			 * can be closed when the region is deleted without
-			 * needing to track if the fd is owned by the framework
-			 * or by the caller.
-			 */
-			guest_memfd = kvm_dup(guest_memfd);
-		}
-
-		region->region.guest_memfd = guest_memfd;
-		region->region.guest_memfd_offset = guest_memfd_offset;
-	} else {
-		region->region.guest_memfd = -1;
-	}
-
 	region->unused_phy_pages = sparsebit_alloc();
 	if (vm_arch_has_protected_memory(vm))
 		region->protected_phy_pages = sparsebit_alloc();
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#120 lore raw

PATCH v10 23/41 KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset}

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Rename local variables and function parameters for the guest memory file descriptor and its offset to use a "gmem_" prefix instead of "guest_memfd_".

No functional change intended.

---
 tools/testing/selftests/kvm/include/kvm_util.h |  6 +++---
 tools/testing/selftests/kvm/lib/kvm_util.c     | 26 +++++++++++++-------------
 2 files changed, 16 insertions(+), 16 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -3
index 5a7a455b53875..cf49a84de02e8 100644
@@ -690,17 +690,17 @@ int __vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags,
 				gpa_t gpa, u64 size, void *hva);
 void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 				gpa_t gpa, u64 size, void *hva,
-				u32 guest_memfd, u64 guest_memfd_offset);
+				u32 gmem_fd, u64 gmem_offset);
 int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 				 gpa_t gpa, u64 size, void *hva,
-				 u32 guest_memfd, u64 guest_memfd_offset);
+				 u32 gmem_fd, u64 gmem_offset);
 
 void vm_userspace_mem_region_add(struct kvm_vm *vm,
 				 enum vm_mem_backing_src_type src_type,
 				 gpa_t gpa, u32 slot, u64 npages, u32 flags);
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		gpa_t gpa, u32 slot, u64 npages, u32 flags,
-		int guest_memfd_fd, u64 guest_memfd_offset);
+		int gmem_fd, u64 gmem_offset);
 
 #ifndef vm_arch_has_protected_memory
 static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
tools/testing/selftests/kvm/lib/kvm_util.c +13 -13
index 3413d45e34faf..af501a27211a2 100644
@@ -970,7 +970,7 @@ void vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags,
 
 int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 				 gpa_t gpa, u64 size, void *hva,
-				 u32 guest_memfd, u64 guest_memfd_offset)
+				 u32 gmem_fd, u64 gmem_offset)
 {
 	struct kvm_userspace_memory_region2 region = {
 		.slot = slot,
@@ -978,8 +978,8 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 		.guest_phys_addr = gpa,
 		.memory_size = size,
 		.userspace_addr = (uintptr_t)hva,
-		.guest_memfd = guest_memfd,
-		.guest_memfd_offset = guest_memfd_offset,
+		.guest_memfd = gmem_fd,
+		.guest_memfd_offset = gmem_offset,
 	};
 
 	TEST_REQUIRE_SET_USER_MEMORY_REGION2();
@@ -989,10 +989,10 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 
 void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 				gpa_t gpa, u64 size, void *hva,
-				u32 guest_memfd, u64 guest_memfd_offset)
+				u32 gmem_fd, u64 gmem_offset)
 {
 	int ret = __vm_set_user_memory_region2(vm, slot, flags, gpa, size, hva,
-					       guest_memfd, guest_memfd_offset);
+					       gmem_fd, gmem_offset);
 
 	TEST_ASSERT(!ret, "KVM_SET_USER_MEMORY_REGION2 failed, errno = %d (%s)",
 		    errno, strerror(errno));
@@ -1002,7 +1002,7 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 /* FIXME: This thing needs to be ripped apart and rewritten. */
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		gpa_t gpa, u32 slot, u64 npages, u32 flags,
-		int guest_memfd, u64 guest_memfd_offset)
+		int gmem_fd, u64 gmem_offset)
 {
 	int ret;
 	struct userspace_mem_region *region;
@@ -1078,12 +1078,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		region->mmap_size += alignment;
 
 	if (flags & KVM_MEM_GUEST_MEMFD) {
-		if (guest_memfd < 0) {
-			u32 guest_memfd_flags = 0;
+		if (gmem_fd < 0) {
+			u32 gmem_flags = 0;
 
-			TEST_ASSERT(!guest_memfd_offset,
+			TEST_ASSERT(!gmem_offset,
 				    "Offset must be zero when creating new guest_memfd");
-			guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags);
+			gmem_fd = vm_create_guest_memfd(vm, mem_size, gmem_flags);
 		} else {
 			/*
 			 * Install a unique fd for each memslot so that the fd
@@ -1091,11 +1091,11 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 			 * needing to track if the fd is owned by the framework
 			 * or by the caller.
 			 */
-			guest_memfd = kvm_dup(guest_memfd);
+			gmem_fd = kvm_dup(gmem_fd);
 		}
 
-		region->region.guest_memfd = guest_memfd;
-		region->region.guest_memfd_offset = guest_memfd_offset;
+		region->region.guest_memfd = gmem_fd;
+		region->region.guest_memfd_offset = gmem_offset;
 	} else {
 		region->region.guest_memfd = -1;
 	}
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#121 lore raw

PATCH v10 24/41 KVM: selftests: Add support for mmap() on guest_memfd in core library

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Accept gmem_flags in vm_mem_add() to be able to create a guest_memfd within vm_mem_add().

When vm_mem_add() is used to set up a guest_memfd for a memslot, set up the provided (or created) gmem_fd as the fd for the user memory region. This makes it available to be mmap()-ed from just like fds from other memory sources.

For guest_memfds, mmap() using gmem_offset instead of 0 all the time.

Always use MAP_SHARED if mmap-ing from guest_memfd instead of reading flag from the configured src_type, which doesn't include guest_memfd.

Add a kvm_slot_to_fd() helper to provide convenient access to the file descriptor of a memslot.

Update existing callers of vm_mem_add() to pass 0 for gmem_flags to preserve existing behavior.

[mmap() with gmem_offset, and always with MAP_SHARED.]
---
 tools/testing/selftests/kvm/include/kvm_util.h     |  7 +++++-
 tools/testing/selftests/kvm/lib/kvm_util.c         | 28 ++++++++++++----------
 .../kvm/x86/private_mem_conversions_test.c         |  2 +-
 3 files changed, 23 insertions(+), 14 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +6 -1
index cf49a84de02e8..a091c495442a1 100644
@@ -700,7 +700,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,
 				 gpa_t gpa, u32 slot, u64 npages, u32 flags);
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		gpa_t gpa, u32 slot, u64 npages, u32 flags,
-		int gmem_fd, u64 gmem_offset);
+		int gmem_fd, u64 gmem_offset, u64 gmem_flags);
 
 #ifndef vm_arch_has_protected_memory
 static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
@@ -732,6 +732,11 @@ void *addr_gva2hva(struct kvm_vm *vm, gva_t gva);
 gpa_t addr_hva2gpa(struct kvm_vm *vm, void *hva);
 void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa);
 
+static inline int kvm_slot_to_fd(struct kvm_vm *vm, u32 slot)
+{
+	return memslot2region(vm, slot)->fd;
+}
+
 #ifndef vcpu_arch_put_guest
 #define vcpu_arch_put_guest(mem, val) do { (mem) = (val); } while (0)
 #endif
tools/testing/selftests/kvm/lib/kvm_util.c +16 -12
index af501a27211a2..08d0bbcd421ea 100644
@@ -1002,12 +1002,14 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags,
 /* FIXME: This thing needs to be ripped apart and rewritten. */
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		gpa_t gpa, u32 slot, u64 npages, u32 flags,
-		int gmem_fd, u64 gmem_offset)
+		int gmem_fd, u64 gmem_offset, u64 gmem_flags)
 {
 	int ret;
 	struct userspace_mem_region *region;
 	size_t backing_src_pagesz = get_backing_src_pagesz(src_type);
+	int mmap_flags = vm_mem_backing_src_alias(src_type)->flag;
 	size_t mem_size = npages * vm->page_size;
+	off_t mmap_offset = 0;
 	size_t alignment = 1;
 
 	TEST_REQUIRE_SET_USER_MEMORY_REGION2();
@@ -1079,8 +1081,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 
 	if (flags & KVM_MEM_GUEST_MEMFD) {
 		if (gmem_fd < 0) {
-			u32 gmem_flags = 0;
-
 			TEST_ASSERT(!gmem_offset,
 				    "Offset must be zero when creating new guest_memfd");
 			gmem_fd = vm_create_guest_memfd(vm, mem_size, gmem_flags);
@@ -1101,13 +1101,17 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 	}
 
 	region->fd = -1;
-	if (backing_src_is_shared(src_type))
+	if (flags & KVM_MEM_GUEST_MEMFD && gmem_flags & GUEST_MEMFD_FLAG_MMAP) {
+		region->fd = kvm_dup(gmem_fd);
+		mmap_flags = MAP_SHARED;
+		mmap_offset = gmem_offset;
+	} else if (backing_src_is_shared(src_type)) {
 		region->fd = kvm_memfd_alloc(region->mmap_size,
 					     src_type == VM_MEM_SRC_SHARED_HUGETLB);
+	}
 
-	region->mmap_start = kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE,
-				      vm_mem_backing_src_alias(src_type)->flag,
-				      region->fd);
+	region->mmap_start = __kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE,
+					mmap_flags, region->fd, mmap_offset);
 
 	TEST_ASSERT(!is_backing_src_hugetlb(src_type) ||
 		    region->mmap_start == align_ptr_up(region->mmap_start, backing_src_pagesz),
@@ -1153,10 +1157,10 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 
 	/* If shared memory, create an alias. */
 	if (region->fd >= 0) {
-		region->mmap_alias = kvm_mmap(region->mmap_size,
-					      PROT_READ | PROT_WRITE,
-					      vm_mem_backing_src_alias(src_type)->flag,
-					      region->fd);
+		region->mmap_alias = __kvm_mmap(region->mmap_size,
+						PROT_READ | PROT_WRITE,
+						mmap_flags, region->fd,
+						mmap_offset);
 
 		/* Align host alias address */
 		region->host_alias = align_ptr_up(region->mmap_alias, alignment);
@@ -1167,7 +1171,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,
 				 enum vm_mem_backing_src_type src_type,
 				 gpa_t gpa, u32 slot, u64 npages, u32 flags)
 {
-	vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0);
+	vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0, 0);
 }
 
 /*
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -1
index 3aa0673f79f33..f3de2a4ab0551 100644
@@ -399,7 +399,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v
 	for (i = 0; i < nr_memslots; i++)
 		vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i,
 			   BASE_DATA_SLOT + i, slot_size / vm->page_size,
-			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i);
+			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, 0);
 
 	for (i = 0; i < nr_vcpus; i++) {
 		gpa_t gpa =  BASE_DATA_GPA + i * per_cpu_size;
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#122 lore raw

PATCH v10 25/41 KVM: selftests: Add selftests global for guest memory attributes capability

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Add a global variable, kvm_has_gmem_attributes, to make the result of checking for KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES available to all tests.

kvm_has_gmem_attributes is true if guest_memfd tracks memory attributes, as opposed to VM-level tracking.

This global variable is synced to the guest for testing convenience, to avoid introducing subtle bugs when host/guest state is desynced.

---
 tools/testing/selftests/kvm/include/test_util.h | 2 ++
 tools/testing/selftests/kvm/lib/kvm_util.c      | 5 +++++
 2 files changed, 7 insertions(+)
tools/testing/selftests/kvm/include/test_util.h +2 -0
index a6a3e16578959..d5825211f2727 100644
@@ -116,6 +116,8 @@ struct kvm_random_state {
 extern u32 kvm_random_seed;
 extern struct kvm_random_state kvm_rng;
 
+extern bool kvm_has_gmem_attributes;
+
 struct kvm_random_state new_kvm_random_state(u32 seed);
 u32 kvm_random_u32(struct kvm_random_state *state);
 
tools/testing/selftests/kvm/lib/kvm_util.c +5 -0
index 08d0bbcd421ea..cd468b2bb8343 100644
@@ -32,6 +32,8 @@ static void kvm_seed_rng(u32 seed)
 	kvm_rng = new_kvm_random_state(kvm_random_seed);
 }
 
+bool kvm_has_gmem_attributes;
+
 static size_t vcpu_mmap_sz(void);
 
 int __open_path_or_exit(const char *path, int flags, const char *enoent_help)
@@ -527,6 +529,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus,
 		kvm_seed_rng(kvm_random_seed);
 
 	sync_global_to_guest(vm, kvm_rng);
+	sync_global_to_guest(vm, kvm_has_gmem_attributes);
 
 	kvm_arch_vm_post_create(vm, nr_runnable_vcpus);
 
@@ -2311,6 +2314,8 @@ void __attribute((constructor)) kvm_selftest_init(void)
 	srandom(time(0));
 	kvm_seed_rng(random());
 
+	kvm_has_gmem_attributes = kvm_has_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES);
+
 	kvm_selftest_arch_init();
 }
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#123 lore raw

PATCH v10 26/41 KVM: selftests: Add helpers for calling ioctls on guest_memfd

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Add helper functions to kvm_util.h to support calling ioctls, specifically KVM_SET_MEMORY_ATTRIBUTES2, on a guest_memfd file descriptor.

Introduce gmem_ioctl() and __gmem_ioctl() macros, modeled after the existing vm_ioctl() helpers, to provide a standard way to call ioctls on a guest_memfd.

Add gmem_set_memory_attributes() and its derivatives (gmem_set_private(), gmem_set_shared()) to set memory attributes on a guest_memfd region. Also provide "__" variants that return the ioctl error code instead of aborting the test. These helpers will be used by upcoming guest_memfd tests.

To avoid code duplication, factor out the check for supported memory attributes into a new macro, TEST_ASSERT_SUPPORTED_ATTRIBUTES, and use it in both the existing vm_set_memory_attributes() and the new gmem_set_memory_attributes() helpers.

---
 tools/testing/selftests/kvm/include/kvm_util.h | 94 +++++++++++++++++++++++---
 1 file changed, 86 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +86 -8
index a091c495442a1..575032e9791d2 100644
@@ -392,6 +392,16 @@ static __always_inline void static_assert_is_vcpu(struct kvm_vcpu *vcpu) { }
 	__TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (vcpu)->vm);	\
 })
 
+#define __gmem_ioctl(gmem_fd, cmd, arg)				\
+	kvm_do_ioctl(gmem_fd, cmd, arg)
+
+#define gmem_ioctl(gmem_fd, cmd, arg)				\
+({								\
+	int ret = __gmem_ioctl(gmem_fd, cmd, arg);		\
+								\
+	TEST_ASSERT(!ret, __KVM_IOCTL_ERROR(#cmd, ret));	\
+})
+
 /*
  * Looks up and returns the value corresponding to the capability
  * (KVM_CAP_*) given by cap.
@@ -418,8 +428,16 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)
 	vm_ioctl(vm, KVM_ENABLE_CAP, &enable_cap);
 }
 
+/*
+ * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes.  These
+ * flows need significant enhancements to support multiple attributes.
+ */
+#define TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes)				\
+	TEST_ASSERT(!(attributes) || (attributes) == KVM_MEMORY_ATTRIBUTE_PRIVATE,	\
+		    "Update me to support multiple attributes!")
+
 static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
-					    u64 size, u64 attributes)
+					    size_t size, u64 attributes)
 {
 	struct kvm_memory_attributes attr = {
 		.attributes = attributes,
@@ -428,17 +446,11 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
 		.flags = 0,
 	};
 
-	/*
-	 * KVM_SET_MEMORY_ATTRIBUTES overwrites _all_ attributes.  These flows
-	 * need significant enhancements to support multiple attributes.
-	 */
-	TEST_ASSERT(!attributes || attributes == KVM_MEMORY_ATTRIBUTE_PRIVATE,
-		    "Update me to support multiple attributes!");
+	TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes);
 
 	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
 }
 
-
 static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa,
 				      u64 size)
 {
@@ -451,6 +463,72 @@ static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa,
 	vm_set_memory_attributes(vm, gpa, size, 0);
 }
 
+static inline int __gmem_set_memory_attributes(int fd, u64 offset,
+					       size_t size, u64 attributes,
+					       u64 *error_offset)
+{
+	struct kvm_memory_attributes2 attr = {
+		.attributes = attributes,
+		.offset = offset,
+		.size = size,
+		.flags = 0,
+		.error_offset = 0,
+	};
+	int r;
+
+	r = __gmem_ioctl(fd, KVM_SET_MEMORY_ATTRIBUTES2, &attr);
+
+	/* Copy error_offset regardless of r so caller can check. */
+	if (error_offset)
+		*error_offset = attr.error_offset;
+
+	return r;
+}
+
+static inline int __gmem_set_private(int fd, u64 offset, size_t size,
+				     u64 *error_offset)
+{
+	return __gmem_set_memory_attributes(fd, offset, size,
+					    KVM_MEMORY_ATTRIBUTE_PRIVATE,
+					    error_offset);
+}
+
+static inline int __gmem_set_shared(int fd, u64 offset, size_t size,
+				    u64 *error_offset)
+{
+	return __gmem_set_memory_attributes(fd, offset, size, 0,
+					    error_offset);
+}
+
+static inline void gmem_set_memory_attributes(int fd, u64 offset,
+					      size_t size, u64 attributes)
+{
+	struct kvm_memory_attributes2 attr = {
+		.attributes = attributes,
+		.offset = offset,
+		.size = size,
+		.flags = 0,
+	};
+
+	TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes);
+
+	__TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) > 0,
+		       "No valid attributes for guest_memfd ioctl!");
+
+	gmem_ioctl(fd, KVM_SET_MEMORY_ATTRIBUTES2, &attr);
+}
+
+static inline void gmem_set_private(int fd, u64 offset, size_t size)
+{
+	gmem_set_memory_attributes(fd, offset, size,
+				   KVM_MEMORY_ATTRIBUTE_PRIVATE);
+}
+
+static inline void gmem_set_shared(int fd, u64 offset, size_t size)
+{
+	gmem_set_memory_attributes(fd, offset, size, 0);
+}
+
 void vm_guest_mem_fallocate(struct kvm_vm *vm, gpa_t gpa, u64 size,
 			    bool punch_hole);
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#124 lore raw

PATCH v10 27/41 KVM: selftests: Test basic single-page conversion flow

In reply to Ackerley Tng

Add a selftest for the guest_memfd memory attribute conversion ioctls. The test starts the guest_memfd as all-private (the default state), and verifies the basic flow of converting a single page to shared and then back to private.

Add infrastructure that supports extensions to other conversion flow tests. This infrastructure will be used in upcoming patches for other conversion tests.

Add test as an x86-specific test since guest_memfd's testing vehicle (KVM_X86_SW_PROTECTED_VM) is x86-specific.

---
v10: Updated __GMEM_CONVERSION_TEST macro to avoid potential macro argument
reuse

---
 tools/testing/selftests/kvm/Makefile.kvm           |   1 +
 .../kvm/x86/guest_memfd_conversions_test.c         | 200 +++++++++++++++++++++
 2 files changed, 201 insertions(+)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 00123169a190a..049306cf0bb7a 100644
@@ -155,6 +155,7 @@ TEST_GEN_PROGS_x86 += x86/max_vcpuid_cap_test
 TEST_GEN_PROGS_x86 += x86/triple_fault_event_test
 TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
 TEST_GEN_PROGS_x86 += x86/aperfmperf_test
+TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test
 TEST_GEN_PROGS_x86 += access_tracking_perf_test
 TEST_GEN_PROGS_x86 += coalesced_io_test
 TEST_GEN_PROGS_x86 += dirty_log_perf_test
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c new file +200 -0

200 lines not on this page · raw message

Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#125 lore raw

PATCH v10 28/41 KVM: selftests: Test conversion flow when INIT_SHARED

In reply to Ackerley Tng

Add a test case to verify that conversions between private and shared memory work correctly when the memory is initially created as shared.

---
 .../selftests/kvm/x86/guest_memfd_conversions_test.c        | 13 +++++++++++++
 1 file changed, 13 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +13 -0
index 77f95d49349ce..4c20d3d8e9349 100644
@@ -97,6 +97,12 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages)		\
 #define GMEM_CONVERSION_TEST_INIT_PRIVATE(test)					\
 	__GMEM_CONVERSION_TEST_INIT_PRIVATE(test, 1)
 
+#define __GMEM_CONVERSION_TEST_INIT_SHARED(test, __nr_pages)			\
+	GMEM_CONVERSION_TEST(test, __nr_pages, GUEST_MEMFD_FLAG_INIT_SHARED)
+
+#define GMEM_CONVERSION_TEST_INIT_SHARED(test)					\
+	__GMEM_CONVERSION_TEST_INIT_SHARED(test, 1)
+
 struct guest_check_data {
 	void *mem;
 	char expected_val;
@@ -188,6 +194,13 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(init_private)
 	test_convert_to_private(t, 0, 'C', 'E');
 }
 
+GMEM_CONVERSION_TEST_INIT_SHARED(init_shared)
+{
+	test_shared(t, 0, 0, 'A', 'B');
+	test_convert_to_private(t, 0, 'B', 'C');
+	test_convert_to_shared(t, 0, 'C', 'D', 'E');
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#126 lore raw

PATCH v10 29/41 KVM: selftests: Test conversion precision in guest_memfd

In reply to Ackerley Tng

The existing guest_memfd conversion tests only use single-page memory regions. This provides no coverage for multi-page guest_memfd objects, specifically whether KVM correctly handles the page index for conversion operations. An incorrect implementation could, for example, always operate on the first page regardless of the index provided.

Add a new test case to verify that conversions between private and shared memory correctly target the specified page within a multi-page guest_memfd.

This test also verifies the precision of memory conversions by converting a single page an then iterating through all other pages ensure they remain in their original state.

To support this test, add a new GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED macro that handles setting up and tearing down the VM for each page iteration. The teardown logic is adjusted to prevent a double-free in this new scenario.

---

v10: Update macro GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED to avoid
potential macro argument reuse

---
 .../kvm/x86/guest_memfd_conversions_test.c         | 67 ++++++++++++++++++++++
 1 file changed, 67 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +67 -0
index 4c20d3d8e9349..a8d89df53e46d 100644
@@ -61,8 +61,13 @@ static void gmem_conversions_do_setup(test_data_t *t, int nr_pages,
 
 static void gmem_conversions_do_teardown(test_data_t *t)
 {
+	/* Use NULL to avoid second free in FIXTURE_TEARDOWN (multipage tests). */
+	if (!t->vcpu)
+		return;
+
 	/* No need to close gmem_fd, it's owned by the VM structure. */
 	kvm_vm_free(t->vcpu->vm);
+	t->vcpu = NULL;
 }
 
 FIXTURE_TEARDOWN(gmem_conversions)
@@ -103,6 +108,30 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages)		\
 #define GMEM_CONVERSION_TEST_INIT_SHARED(test)					\
 	__GMEM_CONVERSION_TEST_INIT_SHARED(test, 1)
 
+/*
+ * Repeats test over nr_pages in a guest_memfd of size nr_pages, providing each
+ * test iteration with test_page, the index of the page under test in
+ * guest_memfd. test_page takes values 0..(nr_pages - 1) inclusive.
+ */
+#define GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(test, __nr_pages)		\
+static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages,	\
+						const int test_page);		\
+										\
+TEST_F(gmem_conversions, test)							\
+{										\
+	const u64 flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED; \
+	const int nr = (__nr_pages);						\
+	int i;									\
+										\
+	for (i = 0; i < nr; ++i) {						\
+		gmem_conversions_do_setup(self, nr, flags);			\
+		__gmem_conversions_multipage_##test(self, nr, i);		\
+		gmem_conversions_do_teardown(self);				\
+	}									\
+}										\
+static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages,	\
+						const int test_page)
+
 struct guest_check_data {
 	void *mem;
 	char expected_val;
@@ -201,6 +230,44 @@ GMEM_CONVERSION_TEST_INIT_SHARED(init_shared)
 	test_convert_to_shared(t, 0, 'C', 'D', 'E');
 }
 
+GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4)
+{
+	int i;
+
+	/* Get a char that varies with both i and n. */
+#define combine(x, n) (((x) << 4) + (n))
+#define i_(n) (combine(i, n))
+#define t_(n) (combine(test_page, n))
+
+	/*
+	 * Start with the highest index, to catch any errors when, perhaps, the
+	 * first page is returned even for the last index.
+	 */
+	for (i = nr_pages - 1; i >= 0; --i)
+		test_shared(t, i, 0, i_(0), i_(2));
+
+	test_convert_to_private(t, test_page, t_(2), t_(3));
+
+	for (i = 0; i < nr_pages; ++i) {
+		if (i == test_page)
+			test_private(t, test_page, t_(3), t_(4));
+		else
+			test_shared(t, i, i_(2), i_(3), i_(4));
+	}
+
+	test_convert_to_shared(t, test_page, t_(4), t_(5), t_(6));
+
+	for (i = 0; i < nr_pages; ++i) {
+		char expected = i == test_page ? t_(6) : i_(4);
+
+		test_shared(t, i, expected, i_(7), i_(8));
+	}
+
+#undef t_
+#undef i_
+#undef combine
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#127 lore raw

PATCH v10 30/41 KVM: selftests: Test conversion before allocation

In reply to Ackerley Tng

Add two test cases to the guest_memfd conversions selftest to cover the scenario where a conversion is requested before any memory has been allocated in the guest_memfd region.

The KVM_SET_MEMORY_ATTRIBUTES2 ioctl can be called on a memory region at any time. If the guest had not yet faulted in any pages for that region, the kernel must record the conversion request and apply the requested state when the pages are eventually allocated.

The new tests cover both conversion directions.

---
 .../selftests/kvm/x86/guest_memfd_conversions_test.c       | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +14 -0
index a8d89df53e46d..f46f1530a94b9 100644
@@ -268,6 +268,20 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4)
 #undef combine
 }
 
+/*
+ * Test that even if there are no folios yet, conversion requests are recorded
+ * in guest_memfd.
+ */
+GMEM_CONVERSION_TEST_INIT_SHARED(before_allocation_shared)
+{
+	test_convert_to_private(t, 0, 0, 'A');
+}
+
+GMEM_CONVERSION_TEST_INIT_PRIVATE(before_allocation_private)
+{
+	test_convert_to_shared(t, 0, 0, 'A', 'B');
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#128 lore raw

PATCH v10 31/41 KVM: selftests: Convert with allocated folios in different layouts

In reply to Ackerley Tng

Add a guest_memfd selftest to verify that memory conversions work correctly with allocated folios in different layouts.

By iterating through which pages are initially faulted, the test covers various layouts of contiguous allocated and unallocated regions, exercising conversion with different range layouts.

---
 .../kvm/x86/guest_memfd_conversions_test.c         | 30 ++++++++++++++++++++++
 1 file changed, 30 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +30 -0
index f46f1530a94b9..acdff2d154046 100644
@@ -282,6 +282,36 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(before_allocation_private)
 	test_convert_to_shared(t, 0, 0, 'A', 'B');
 }
 
+/*
+ * Test that when some of the folios in the conversion range are allocated,
+ * conversion requests are handled correctly in guest_memfd.  Vary the ranges
+ * allocated before conversion, using test_page, to cover various layouts of
+ * contiguous allocated and unallocated regions.
+ */
+GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8)
+{
+	const int second_page_to_fault = 4;
+	int i;
+
+	/*
+	 * Fault 2 of the pages to test filemap range operations except when
+	 * test_page == second_page_to_fault.
+	 */
+	host_do_rmw(t->mem, test_page, 0, 'A');
+	if (test_page != second_page_to_fault)
+		host_do_rmw(t->mem, second_page_to_fault, 0, 'A');
+
+	gmem_set_private(t->gmem_fd, 0, nr_pages * page_size);
+	for (i = 0; i < nr_pages; ++i) {
+		char expected = (i == test_page || i == second_page_to_fault) ? 'A' : 0;
+
+		test_private(t, i, expected, 'B');
+	}
+
+	for (i = 0; i < nr_pages; ++i)
+		test_convert_to_shared(t, i, 'B', 'C', 'D');
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#129 lore raw

PATCH v10 32/41 KVM: selftests: Test that truncation does not change shared/private status

In reply to Ackerley Tng

Add a test to verify that deallocating a page in a guest memfd region via fallocate() with FALLOC_FL_PUNCH_HOLE does not alter the shared or private status of the corresponding memory range.

When a page backing a guest memfd mapping is deallocated, e.g., by punching a hole or truncating the file, and then subsequently faulted back in, the new page must inherit the correct shared/private status tracked by guest_memfd.

---
 .../selftests/kvm/x86/guest_memfd_conversions_test.c       | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +14 -0
index acdff2d154046..b7a54d2a90928 100644
@@ -10,6 +10,7 @@
 #include <linux/sizes.h>
 
 #include "kvm_util.h"
+#include "kvm_syscalls.h"
 #include "kselftest_harness.h"
 #include "test_util.h"
 #include "ucall_common.h"
@@ -312,6 +313,19 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8)
 		test_convert_to_shared(t, i, 'B', 'C', 'D');
 }
 
+/* Truncation should not affect shared/private status. */
+GMEM_CONVERSION_TEST_INIT_SHARED(truncate)
+{
+	host_do_rmw(t->mem, 0, 0, 'A');
+	kvm_fallocate(t->gmem_fd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, page_size);
+	host_do_rmw(t->mem, 0, 0, 'A');
+
+	test_convert_to_private(t, 0, 'A', 'B');
+
+	kvm_fallocate(t->gmem_fd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, page_size);
+	test_private(t, 0, 0, 'A');
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#130 lore raw

PATCH v10 33/41 KVM: selftests: Test that shared/private status is consistent across processes

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Add a test to verify that a guest_memfd's shared/private status is consistent across processes, and that any shared pages previously mapped in any process are unmapped from all processes.

The test forks a child process after creating the shared guest_memfd region so that the second process exists alongside the main process for the entire test.

The processes then take turns to access memory to check that the shared/private status is consistent across processes.

---
 .../kvm/x86/guest_memfd_conversions_test.c         | 118 +++++++++++++++++++++
 1 file changed, 118 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +118 -0
index b7a54d2a90928..a972a09c5c6e5 100644
@@ -2,7 +2,9 @@
 /*
  * Copyright (c) 2024, Google LLC.
  */
+#include <pthread.h>
 #include <sys/mman.h>
+#include <time.h>
 #include <unistd.h>
 
 #include <linux/align.h>
@@ -326,6 +328,122 @@ GMEM_CONVERSION_TEST_INIT_SHARED(truncate)
 	test_private(t, 0, 0, 'A');
 }
 
+/* Test that shared/private memory protections work and are seen from any process. */
+GMEM_CONVERSION_TEST_INIT_SHARED(forked_accesses)
+{
+	enum test_state {
+		STATE_INIT,
+		STATE_CHECK_SHARED,
+		STATE_DONE_CHECKING_SHARED,
+		STATE_CHECK_PRIVATE,
+		STATE_DONE_CHECKING_PRIVATE,
+	};
+
+	struct sync_state {
+		pthread_mutex_t mutex;
+		pthread_cond_t cond;
+		enum test_state step;
+	} *sync;
+
+	pthread_mutexattr_t mattr;
+	pthread_condattr_t cattr;
+	pid_t child_pid, parent_pid;
+	int status;
+
+	sync = kvm_mmap(sizeof(*sync), PROT_READ | PROT_WRITE,
+			MAP_SHARED | MAP_ANONYMOUS, -1);
+
+	pthread_mutexattr_init(&mattr);
+	pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
+	pthread_mutex_init(&sync->mutex, &mattr);
+	pthread_mutexattr_destroy(&mattr);
+
+	pthread_condattr_init(&cattr);
+	pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED);
+	pthread_cond_init(&sync->cond, &cattr);
+	pthread_condattr_destroy(&cattr);
+
+	sync->step = STATE_INIT;
+
+#define TEST_STATE_AWAIT(__state)						\
+	do {									\
+		pthread_mutex_lock(&sync->mutex);				\
+		while (sync->step != (__state)) {				\
+			struct timespec ts, stop;				\
+			int ret;						\
+										\
+			clock_gettime(CLOCK_REALTIME, &ts);			\
+			stop = timespec_add_ns(ts, 100 * 1000000UL);		\
+										\
+			ret = pthread_cond_timedwait(&sync->cond, &sync->mutex, &stop); \
+			if (ret == ETIMEDOUT) {					\
+				bool alive = (child_pid == 0) ?			\
+					     (getppid() == parent_pid) :		\
+					     (waitpid(child_pid, NULL, WNOHANG) == 0); \
+				TEST_ASSERT(alive, "Other process exited prematurely"); \
+			} else {						\
+				TEST_ASSERT(!ret, "pthread_cond_timedwait failed"); \
+			}							\
+		}								\
+		pthread_mutex_unlock(&sync->mutex);				\
+	} while (0)
+
+#define TEST_STATE_SET(__state)							\
+	do {									\
+		pthread_mutex_lock(&sync->mutex);				\
+		sync->step = (__state);						\
+		pthread_cond_broadcast(&sync->cond);				\
+		pthread_mutex_unlock(&sync->mutex);				\
+	} while (0)
+
+	parent_pid = getpid();
+	child_pid = fork();
+	TEST_ASSERT(child_pid != -1, "fork failed");
+
+	if (child_pid == 0) {
+		const char inconsequential = 0xdd;
+
+		TEST_STATE_AWAIT(STATE_CHECK_SHARED);
+
+		/*
+		 * This maps the pages into the child process as well, and tests
+		 * that the conversion process will unmap the guest_memfd memory
+		 * from all processes.
+		 */
+		host_do_rmw(t->mem, 0, 0xB, 0xC);
+
+		TEST_STATE_SET(STATE_DONE_CHECKING_SHARED);
+		TEST_STATE_AWAIT(STATE_CHECK_PRIVATE);
+
+		TEST_EXPECT_SIGBUS(READ_ONCE(t->mem[0]));
+		TEST_EXPECT_SIGBUS(WRITE_ONCE(t->mem[0], inconsequential));
+
+		TEST_STATE_SET(STATE_DONE_CHECKING_PRIVATE);
+		exit(0);
+	}
+
+	test_shared(t, 0, 0, 0xA, 0xB);
+
+	TEST_STATE_SET(STATE_CHECK_SHARED);
+	TEST_STATE_AWAIT(STATE_DONE_CHECKING_SHARED);
+
+	test_convert_to_private(t, 0, 0xC, 0xD);
+
+	TEST_STATE_SET(STATE_CHECK_PRIVATE);
+	TEST_STATE_AWAIT(STATE_DONE_CHECKING_PRIVATE);
+
+	TEST_ASSERT_EQ(waitpid(child_pid, &status, 0), child_pid);
+	TEST_ASSERT(WIFEXITED(status) && WEXITSTATUS(status) == 0,
+		    "Child exited with unexpected status");
+
+	pthread_mutex_destroy(&sync->mutex);
+	pthread_cond_destroy(&sync->cond);
+	kvm_munmap(sync, sizeof(*sync));
+
+#undef TEST_STATE_SET
+#undef TEST_STATE_AWAIT
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#131 lore raw

PATCH v10 34/41 KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST

In reply to Ackerley Tng

Add helper functions to allow KVM selftests to pin memory using CONFIG_GUP_TEST. This is useful for testing scenarios where some page has an increased refcount. such as in guest_memfd in-place conversion tests.

The helpers open /sys/kernel/debug/gup_test and invoke the PIN_LONGTERM_TEST_START and PIN_LONGTERM_TEST_STOP ioctls. Since this functionality depends on the kernel being built with CONFIG_GUP_TEST, provide stub implementations that trigger a test failure if the configuration is missing.

---
 tools/testing/selftests/kvm/include/kvm_util.h |  3 +++
 tools/testing/selftests/kvm/lib/kvm_util.c     | 25 +++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
index 575032e9791d2..3cfe243d7c8b0 100644
@@ -1206,6 +1206,9 @@ static inline int pin_self_to_any_cpu(void)
 	return pin_task_to_any_cpu(pthread_self());
 }
 
+void pin_pages(void *vaddr, uint64_t size);
+void unpin_pages(void);
+
 void kvm_print_vcpu_pinning_help(void);
 void kvm_parse_vcpu_pinning(const char *pcpus_string, u32 vcpu_to_pcpu[],
 			    int nr_vcpus);
tools/testing/selftests/kvm/lib/kvm_util.c +25 -0
index cd468b2bb8343..b1456ed76e27e 100644
@@ -19,6 +19,8 @@
 #include <unistd.h>
 #include <linux/kernel.h>
 
+#include "../../../../../mm/gup_test.h"
+
 #define KVM_UTIL_MIN_PFN	2
 
 u32 kvm_random_seed;
@@ -645,6 +647,29 @@ int __pin_task_to_cpu(pthread_t task, int cpu)
 	return pthread_setaffinity_np(task, sizeof(cpuset), &cpuset);
 }
 
+static int gup_test_fd = -1;
+
+void pin_pages(void *vaddr, uint64_t size)
+{
+	const struct pin_longterm_test args = {
+		.addr = (uint64_t)vaddr,
+		.size = size,
+		.flags = PIN_LONGTERM_TEST_FLAG_USE_WRITE,
+	};
+
+	gup_test_fd = __open_path_or_exit("/sys/kernel/debug/gup_test", O_RDWR,
+					  "Is CONFIG_GUP_TEST enabled?");
+
+	TEST_ASSERT_EQ(ioctl(gup_test_fd, PIN_LONGTERM_TEST_START, &args), 0);
+}
+
+void unpin_pages(void)
+{
+	TEST_ASSERT_EQ(ioctl(gup_test_fd, PIN_LONGTERM_TEST_STOP), 0);
+
+	kvm_free_fd(gup_test_fd);
+}
+
 static u32 parse_pcpu(const char *cpu_str, const cpu_set_t *allowed_mask)
 {
 	u32 pcpu = atoi_non_negative("CPU number", cpu_str);
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#132 lore raw

PATCH v10 35/41 KVM: selftests: Test conversion with elevated page refcount

In reply to Ackerley Tng

Add a selftest to verify that converting a shared guest_memfd page to a private page fails if the page has an elevated reference count.

When KVM converts a shared page to a private one, it expects the page to have a reference count equal to the reference counts taken by the filemap. If another kernel subsystem holds a reference to the page, the conversion must be aborted.

The test asserts that both bulk and single-page conversion attempts correctly fail with EAGAIN for the pinned page. After the page is unpinned, the test verifies that subsequent conversions succeed.

---
 .../kvm/x86/guest_memfd_conversions_test.c         | 56 ++++++++++++++++++++++
 1 file changed, 56 insertions(+)
tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +56 -0
index a972a09c5c6e5..e1c84df0f0596 100644
@@ -444,6 +444,62 @@ GMEM_CONVERSION_TEST_INIT_SHARED(forked_accesses)
 #undef TEST_STATE_AWAIT
 }
 
+static void test_convert_to_private_fails(test_data_t *t, u64 pgoff,
+					  size_t nr_pages,
+					  u64 expected_error_offset)
+{
+	/* +1 to make it anything but expected_error_offset. */
+	u64 error_offset = expected_error_offset + 1;
+	u64 offset = pgoff * page_size;
+	int ret;
+
+	do {
+		ret = __gmem_set_private(t->gmem_fd, offset,
+					 nr_pages * page_size, &error_offset);
+	} while (ret == -1 && errno == EINTR);
+	TEST_ASSERT(ret == -1 && errno == EAGAIN,
+		    "Wanted EAGAIN on page %lu, got %d (ret = %d)", pgoff,
+		    errno, ret);
+	TEST_ASSERT_EQ(error_offset, expected_error_offset);
+}
+
+GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(elevated_refcount, 4)
+{
+	int i;
+
+	pin_pages(t->mem + test_page * page_size, page_size);
+
+	for (i = 0; i < nr_pages; i++)
+		test_shared(t, i, 0, 'A', 'B');
+
+	/*
+	 * Converting in bulk should fail as long any page in the range has
+	 * unexpected refcounts.
+	 */
+	test_convert_to_private_fails(t, 0, nr_pages, test_page * page_size);
+
+	for (i = 0; i < nr_pages; i++) {
+		/*
+		 * Converting page-wise should also fail as long any page in the
+		 * range has unexpected refcounts.
+		 */
+		if (i == test_page)
+			test_convert_to_private_fails(t, i, 1, test_page * page_size);
+		else
+			test_convert_to_private(t, i, 'B', 'C');
+	}
+
+	unpin_pages();
+
+	gmem_set_private(t->gmem_fd, 0, nr_pages * page_size);
+
+	for (i = 0; i < nr_pages; i++) {
+		char expected = i == test_page ? 'B' : 'C';
+
+		test_private(t, i, expected, 'D');
+	}
+}
+
 int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#133 lore raw

PATCH v10 36/41 KVM: selftests: Reset shared memory after hole-punching

In reply to Ackerley Tng

private_mem_conversions_test used to reset the shared memory that was used for the test to an initial pattern at the end of each test iteration. Then, it would punch out the pages, which would zero memory.

Without in-place conversion, the resetting would write shared memory, and hole-punching will zero private memory, hence resetting the test to the state at the beginning of the for loop.

With in-place conversion, resetting writes memory as shared, and hole-punching zeroes the same physical memory, hence undoing the reset done before the hole punch.

Move the resetting after the hole-punching, and reset the entire PER_CPU_DATA_SIZE instead of just the tested range.

With in-place conversion, this zeroes and then resets the same physical memory. Without in-place conversion, the private memory is zeroed, and the shared memory is reset to init_p.

This is sufficient since at each test stage, the memory is assumed to start as shared, and private memory is always assumed to start zeroed. Conversion zeroes memory, so the future test stages will work as expected.

Fixes 43f623f350ce1 ("KVM: selftests: Add x86-only selftest for private memory conversions")
---
 tools/testing/selftests/kvm/x86/private_mem_conversions_test.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +6 -3
index f3de2a4ab0551..cf50e9a332c5b 100644
@@ -202,15 +202,18 @@ static void guest_test_explicit_conversion(u64 base_gpa, bool do_fallocate)
 		guest_sync_shared(gpa, size, p3, p4);
 		memcmp_g(gpa, p4, size);
 
-		/* Reset the shared memory back to the initial pattern. */
-		memset((void *)gpa, init_p, size);
-
 		/*
 		 * Free (via PUNCH_HOLE) *all* private memory so that the next
 		 * iteration starts from a clean slate, e.g. with respect to
 		 * whether or not there are pages/folios in guest_mem.
 		 */
 		guest_map_shared(base_gpa, PER_CPU_DATA_SIZE, true);
+
+		/*
+		 * Hole-punching above zeroed private memory. Reset shared
+		 * memory in preparation for the next GUEST_STAGE.
+		 */
+		memset((void *)base_gpa, init_p, PER_CPU_DATA_SIZE);
 	}
 }
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#134 lore raw

PATCH v10 37/41 KVM: selftests: Provide function to look up guest_memfd details from gpa

In reply to Ackerley Tng

Introduce a new helper, kvm_gpa_to_guest_memfd(), to find the guest_memfd-related details of a memory region that contains a given guest physical address (GPA).

The function returns the file descriptor for the memfd, the offset into the file that corresponds to the GPA, and the number of bytes remaining in the region from that GPA.

kvm_gpa_to_guest_memfd() was factored out from vm_guest_mem_fallocate(); refactor vm_guest_mem_fallocate() to use the new helper.

---
 tools/testing/selftests/kvm/include/kvm_util.h |  3 +++
 tools/testing/selftests/kvm/lib/kvm_util.c     | 37 ++++++++++++++++----------
 2 files changed, 26 insertions(+), 14 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
index 3cfe243d7c8b0..795c9eb311071 100644
@@ -428,6 +428,9 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)
 	vm_ioctl(vm, KVM_ENABLE_CAP, &enable_cap);
 }
 
+int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset,
+			   size_t *nr_bytes);
+
 /*
  * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes.  These
  * flows need significant enhancements to support multiple attributes.
tools/testing/selftests/kvm/lib/kvm_util.c +23 -14
index b1456ed76e27e..1188d0a6c0a5f 100644
@@ -1332,27 +1332,20 @@ void vm_guest_mem_fallocate(struct kvm_vm *vm, u64 base, u64 size,
 			    bool punch_hole)
 {
 	const int mode = FALLOC_FL_KEEP_SIZE | (punch_hole ? FALLOC_FL_PUNCH_HOLE : 0);
-	struct userspace_mem_region *region;
 	u64 end = base + size;
-	gpa_t gpa, len;
 	off_t fd_offset;
-	int ret;
+	int fd, ret;
+	size_t len;
+	gpa_t gpa;
 
 	for (gpa = base; gpa < end; gpa += len) {
-		u64 offset;
-
-		region = userspace_mem_region_find(vm, gpa, gpa);
-		TEST_ASSERT(region && region->region.flags & KVM_MEM_GUEST_MEMFD,
-			    "Private memory region not found for GPA 0x%lx", gpa);
+		fd = kvm_gpa_to_guest_memfd(vm, gpa, &fd_offset, &len);
+		len = min(end - gpa, len);
 
-		offset = gpa - region->region.guest_phys_addr;
-		fd_offset = region->region.guest_memfd_offset + offset;
-		len = min_t(u64, end - gpa, region->region.memory_size - offset);
-
-		ret = fallocate(region->region.guest_memfd, mode, fd_offset, len);
+		ret = fallocate(fd, mode, fd_offset, len);
 		TEST_ASSERT(!ret, "fallocate() failed to %s at %lx (len = %lu), fd = %d, mode = %x, offset = %lx",
 			    punch_hole ? "punch hole" : "allocate", gpa, len,
-			    region->region.guest_memfd, mode, fd_offset);
+			    fd, mode, fd_offset);
 	}
 }
 
@@ -1689,6 +1682,22 @@ void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa)
 	return (void *) ((uintptr_t) region->host_alias + offset);
 }
 
+int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset,
+			   size_t *nr_bytes)
+{
+	struct userspace_mem_region *region;
+	gpa_t gpa_offset;
+
+	region = userspace_mem_region_find(vm, gpa, gpa);
+	TEST_ASSERT(region && region->region.flags & KVM_MEM_GUEST_MEMFD,
+		    "guest_memfd memory region not found for GPA 0x%lx", gpa);
+
+	gpa_offset = gpa - region->region.guest_phys_addr;
+	*fd_offset = region->region.guest_memfd_offset + gpa_offset;
+	*nr_bytes = region->region.memory_size - gpa_offset;
+	return region->region.guest_memfd;
+}
+
 /* Create an interrupt controller chip for the specified VM. */
 void vm_create_irqchip(struct kvm_vm *vm)
 {
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#135 lore raw

PATCH v10 38/41 KVM: selftests: Provide common function to set memory attributes

In reply to Ackerley Tng

From: Sean Christopherson <[email protected]>

Introduce vm_mem_set_memory_attributes(), which handles setting of memory attributes for a range of guest physical addresses, regardless of whether the attributes should be set via guest_memfd or via the memory attributes at the VM level.

Refactor existing vm_mem_set_{shared,private} functions to use the new function. Opportunistically update the size parameter to use size_t instead of u64.

---
 tools/testing/selftests/kvm/include/kvm_util.h | 46 +++++++++++++++++++-------
 1 file changed, 34 insertions(+), 12 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +34 -12
index 795c9eb311071..777fa3dbf88d6 100644
@@ -454,18 +454,6 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
 	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
 }
 
-static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa,
-				      u64 size)
-{
-	vm_set_memory_attributes(vm, gpa, size, KVM_MEMORY_ATTRIBUTE_PRIVATE);
-}
-
-static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa,
-				     u64 size)
-{
-	vm_set_memory_attributes(vm, gpa, size, 0);
-}
-
 static inline int __gmem_set_memory_attributes(int fd, u64 offset,
 					       size_t size, u64 attributes,
 					       u64 *error_offset)
@@ -532,6 +520,40 @@ static inline void gmem_set_shared(int fd, u64 offset, size_t size)
 	gmem_set_memory_attributes(fd, offset, size, 0);
 }
 
+static inline void vm_mem_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
+						size_t size, u64 attrs)
+{
+	if (kvm_has_gmem_attributes) {
+		gpa_t end = gpa + size;
+		off_t fd_offset;
+		gpa_t addr;
+		size_t len;
+		int fd;
+
+		for (addr = gpa; addr < end; addr += len) {
+			fd = kvm_gpa_to_guest_memfd(vm, addr, &fd_offset, &len);
+			len = min(end - addr, len);
+
+			gmem_set_memory_attributes(fd, fd_offset, len, attrs);
+		}
+	} else {
+		vm_set_memory_attributes(vm, gpa, size, attrs);
+	}
+}
+
+static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa,
+				      size_t size)
+{
+	vm_mem_set_memory_attributes(vm, gpa, size,
+				     KVM_MEMORY_ATTRIBUTE_PRIVATE);
+}
+
+static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa,
+				     size_t size)
+{
+	vm_mem_set_memory_attributes(vm, gpa, size, 0);
+}
+
 void vm_guest_mem_fallocate(struct kvm_vm *vm, gpa_t gpa, u64 size,
 			    bool punch_hole);
 
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#136 lore raw

PATCH v10 39/41 KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe

In reply to Ackerley Tng

The TEST_EXPECT_SIGBUS macro is not thread-safe as it uses a global sigjmp_buf and installs a global SIGBUS signal handler. If multiple threads execute the macro concurrently, they will race on installing the signal handler and stomp on other threads' jump buffers, leading to incorrect test behavior.

Make TEST_EXPECT_SIGBUS thread-safe with the following changes:

Share the KVM tests' global signal handler. sigaction() applies to all threads; without sharing a global signal handler, one thread may have removed the signal handler that another thread added, hence leading to unexpected signals.

The alternative of layering signal handlers was considered, but calling sigaction() within TEST_EXPECT_SIGBUS() necessarily creates a race. To avoid adding new setup and teardown routines to do sigaction() and keep usage of TEST_EXPECT_SIGBUS() simple, share the KVM tests' global signal handler.

Opportunistically rename report_unexpected_signal to catchall_signal_handler.

To continue to only expect SIGBUS within specific regions of code, use a thread-specific variable, expecting_sigbus, to replace installing and removing signal handlers.

Make the execution environment for the thread, sigjmp_buf, a thread-specific variable.

As part of TEST_EXPECT_SIGBUS(), assert the prerequisite for this setup, that the current signal handler is the catchall_signal_handler.

---
 tools/testing/selftests/kvm/include/test_util.h | 32 +++++++++++++------------
 tools/testing/selftests/kvm/lib/kvm_util.c      | 18 ++++++++++----
 tools/testing/selftests/kvm/lib/test_util.c     |  7 ------
 3 files changed, 30 insertions(+), 27 deletions(-)
tools/testing/selftests/kvm/include/test_util.h +17 -15
index d5825211f2727..75e98ec14107f 100644
@@ -83,21 +83,23 @@ do {									\
 	__builtin_unreachable(); \
 } while (0)
 
-extern sigjmp_buf expect_sigbus_jmpbuf;
-void expect_sigbus_handler(int signum);
-
-#define TEST_EXPECT_SIGBUS(action)						\
-do {										\
-	struct sigaction sa_old, sa_new = {					\
-		.sa_handler = expect_sigbus_handler,				\
-	};									\
-										\
-	sigaction(SIGBUS, &sa_new, &sa_old);					\
-	if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) {				\
-		action;								\
-		TEST_FAIL("'%s' should have triggered SIGBUS", #action);	\
-	}									\
-	sigaction(SIGBUS, &sa_old, NULL);					\
+extern __thread sigjmp_buf expect_sigbus_jmpbuf;
+extern __thread volatile sig_atomic_t expecting_sigbus;
+void catchall_signal_handler(int signum);
+
+#define TEST_EXPECT_SIGBUS(action)					\
+do {									\
+	struct sigaction __sa = {};					\
+									\
+	TEST_ASSERT_EQ(sigaction(SIGBUS, NULL, &__sa), 0);		\
+	TEST_ASSERT_EQ(__sa.sa_handler, &catchall_signal_handler);	\
+									\
+	expecting_sigbus = true;					\
+	if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) {			\
+		action;							\
+		TEST_FAIL("'%s' should have triggered SIGBUS", #action);\
+	}								\
+	expecting_sigbus = false;					\
 } while (0)
 
 size_t parse_size(const char *size);
tools/testing/selftests/kvm/lib/kvm_util.c +13 -5
index 1188d0a6c0a5f..cac5624e99774 100644
@@ -2316,13 +2316,20 @@ __weak void kvm_selftest_arch_init(void)
 {
 }
 
-static void report_unexpected_signal(int signum)
+__thread sigjmp_buf expect_sigbus_jmpbuf;
+__thread volatile sig_atomic_t expecting_sigbus;
+
+void catchall_signal_handler(int signum)
 {
+	switch (signum) {
+	case SIGBUS: {
+		if (expecting_sigbus)
+			siglongjmp(expect_sigbus_jmpbuf, 1);
+
+		TEST_FAIL("Unexpected SIGBUS (%d)\n", signum);
+	}
 #define KVM_CASE_SIGNUM(sig)					\
 	case sig: TEST_FAIL("Unexpected " #sig " (%d)\n", signum)
-
-	switch (signum) {
-	KVM_CASE_SIGNUM(SIGBUS);
 	KVM_CASE_SIGNUM(SIGSEGV);
 	KVM_CASE_SIGNUM(SIGILL);
 	KVM_CASE_SIGNUM(SIGFPE);
@@ -2334,12 +2341,13 @@ static void report_unexpected_signal(int signum)
 void __attribute((constructor)) kvm_selftest_init(void)
 {
 	struct sigaction sig_sa = {
-		.sa_handler = report_unexpected_signal,
+		.sa_handler = catchall_signal_handler,
 	};
 
 	/* Tell stdout not to buffer its content. */
 	setbuf(stdout, NULL);
 
+	expecting_sigbus = false;
 	sigaction(SIGBUS, &sig_sa, NULL);
 	sigaction(SIGSEGV, &sig_sa, NULL);
 	sigaction(SIGILL, &sig_sa, NULL);
tools/testing/selftests/kvm/lib/test_util.c +0 -7
index 4dff72f6bd346..89d6d1b572959 100644
@@ -19,13 +19,6 @@
 #include "test_util.h"
 #include "kvm_syscalls.h"
 
-sigjmp_buf expect_sigbus_jmpbuf;
-
-void __attribute__((used)) expect_sigbus_handler(int signum)
-{
-	siglongjmp(expect_sigbus_jmpbuf, 1);
-}
-
 /*
  * Random number generator that is usable from guest code. This is the
  * Park-Miller LCG using standard constants.
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#137 lore raw

PATCH v10 40/41 KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

In reply to Ackerley Tng

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

2. With in-place conversions, the host's mapping points directly to the guest's memory. When the guest converts a region to private, host access to that region is blocked. Update the test to expect a SIGBUS when attempting to access the host virtual address (HVA) of private memory.

3. Use vm_mem_set_memory_attributes(), which chooses how to set memory attributes based on whether kvm_has_gmem_attributes.

Restrict the test to using VM_MEM_SRC_SHMEM because guest_memfd's required mmap() flags and page sizes happens to align with those of VM_MEM_SRC_SHMEM. As long as VM_MEM_SRC_SHMEM is used for src_type, vm_mem_add() works as intended.

---
 .../kvm/x86/private_mem_conversions_test.c         | 44 ++++++++++++++++++----
 1 file changed, 36 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +36 -8
index cf50e9a332c5b..2af8d41963816 100644
@@ -306,9 +306,12 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu)
 	if (do_fallocate)
 		vm_guest_mem_fallocate(vm, gpa, size, map_shared);
 
-	if (set_attributes)
-		vm_set_memory_attributes(vm, gpa, size,
-					 map_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE);
+	if (set_attributes) {
+		u64 attrs = map_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+		vm_mem_set_memory_attributes(vm, gpa, size, attrs);
+	}
+
 	run->hypercall.ret = 0;
 }
 
@@ -352,8 +355,20 @@ static void *__test_mem_conversions(void *__vcpu)
 				size_t nr_bytes = min_t(size_t, vm->page_size, size - i);
 				u8 *hva = addr_gpa2hva(vm, gpa + i);
 
-				/* In all cases, the host should observe the shared data. */
-				memcmp_h(hva, gpa + i, uc.args[3], nr_bytes);
+				/*
+				 * When using per-guest_memfd memory attributes,
+				 * i.e. in-place conversion, host accesses will
+				 * point at guest memory and should SIGBUS when
+				 * guest memory is private.  When using per-VM
+				 * attributes, i.e. separate backing for shared
+				 * vs. private, the host should always observe
+				 * the shared data.
+				 */
+				if (kvm_has_gmem_attributes &&
+				    uc.args[0] == SYNC_PRIVATE)
+					TEST_EXPECT_SIGBUS(READ_ONCE(*hva));
+				else
+					memcmp_h(hva, gpa + i, uc.args[3], nr_bytes);
 
 				/* For shared, write the new pattern to guest memory. */
 				if (uc.args[0] == SYNC_SHARED)
@@ -382,6 +397,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v
 	const size_t slot_size = memfd_size / nr_memslots;
 	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 	pthread_t threads[KVM_MAX_VCPUS];
+	u64 gmem_flags;
 	struct kvm_vm *vm;
 	int memfd, i;
 
@@ -397,12 +413,17 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v
 
 	vm_enable_cap(vm, KVM_CAP_EXIT_HYPERCALL, (1 << KVM_HC_MAP_GPA_RANGE));
 
-	memfd = vm_create_guest_memfd(vm, memfd_size, 0);
+	if (kvm_has_gmem_attributes)
+		gmem_flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED;
+	else
+		gmem_flags = 0;
+
+	memfd = vm_create_guest_memfd(vm, memfd_size, gmem_flags);
 
 	for (i = 0; i < nr_memslots; i++)
 		vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i,
 			   BASE_DATA_SLOT + i, slot_size / vm->page_size,
-			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, 0);
+			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, gmem_flags);
 
 	for (i = 0; i < nr_vcpus; i++) {
 		gpa_t gpa =  BASE_DATA_GPA + i * per_cpu_size;
@@ -452,17 +473,24 @@ static void usage(const char *cmd)
 
 int main(int argc, char *argv[])
 {
-	enum vm_mem_backing_src_type src_type = DEFAULT_VM_MEM_SRC;
+	enum vm_mem_backing_src_type src_type;
 	u32 nr_memslots = 1;
 	u32 nr_vcpus = 1;
 	int opt;
 
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
 
+	src_type = kvm_has_gmem_attributes ? VM_MEM_SRC_SHMEM :
+					     DEFAULT_VM_MEM_SRC;
+
 	while ((opt = getopt(argc, argv, "hm:s:n:")) != -1) {
 		switch (opt) {
 		case 's':
 			src_type = parse_backing_src_type(optarg);
+			TEST_ASSERT(!kvm_has_gmem_attributes ||
+				    src_type == VM_MEM_SRC_SHMEM,
+				    "Testing in-place conversions, only %s mem_type supported\n",
+				    vm_mem_backing_src_alias(VM_MEM_SRC_SHMEM)->name);
 			break;
 		case 'n':
 			nr_vcpus = atoi_positive("nr_vcpus", optarg);
Signature · 1 line
2.55.0.654.g21b8a5bc05-goog
#138 lore raw

Re: [PATCH v10 40/41] KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

Quoted 11 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng<[email protected]>

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

If I understand correctly, even with in-place gmem, the model that shared memory comes from the userspace_addr and private memory comes from gmem can still work without passing GUEST_MEMFD_FLAG_MMAP flag.

Since this model is not prohibited by KVM. I think we should keep the testcase for this model to ensure test coverage?

#139 lore raw

Re: [PATCH v10 40/41] KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

Quoted 11 lines · Xiaoyao Li
Quoted 10 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng<[email protected]>

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

Quoted 8 lines
Quoted 1 line

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

If I understand correctly, even with in-place gmem, the model that shared memory comes from the userspace_addr and private memory comes from gmem can still work without passing GUEST_MEMFD_FLAG_MMAP flag.

Since this model is not prohibited by KVM. I think we should keep the testcase for this model to ensure test coverage?

+1. Deprecated doesn't mean unsupported.

#140 lore raw

Re: [PATCH v10 40/41] KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

Quoted 14 lines · Sean Christopherson
Quoted 13 lines · Xiaoyao Li
Quoted 11 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng<[email protected]>

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

Quoted 8 lines
Quoted 6 lines

If I understand correctly, even with in-place gmem, the model that shared memory comes from the userspace_addr and private memory comes from gmem can still work without passing GUEST_MEMFD_FLAG_MMAP flag.

Since this model is not prohibited by KVM. I think we should keep the testcase for this model to ensure test coverage?

+1. Deprecated doesn't mean unsupported.

The original test case was for non-in-place conversion, which wasn't removed, it still passes.

Is the ask to also test, under kvm_has_gmem_attributes aka gmem_in_place_conversion, to have guest_memfd used as purely for private memory and have shared memory taken from somewhere else completely?

I can rephrase the changelog and change the code to clarify that this patch is meant to specifically test in-place conversion with MMAP, the intended usage of in-place conversions.

The ask to test gmem for private and something else for shared seems like a different test that can be added separately, perhaps in a follow-up series?

#141 lore raw

Re: [PATCH v10 40/41] KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

Quoted 20 lines · Ackerley Tng
Quoted 18 lines · Sean Christopherson
Quoted 17 lines · Xiaoyao Li
Quoted 11 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng<[email protected]>

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

If I understand correctly, even with in-place gmem, the model that shared memory comes from the userspace_addr and private memory comes from gmem can still work without passing GUEST_MEMFD_FLAG_MMAP flag.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Since this model is not prohibited by KVM. I think we should keep the testcase for this model to ensure test coverage?

+1. Deprecated doesn't mean unsupported.

The original test case was for non-in-place conversion, which wasn't removed, it still passes.

It only passes when

/sys/module/kvm/parameters/gmem_in_place_conversion=n

It fails as below when gmem_in_place_conversion=y

./tools/testing/selftests/kvm/x86/private_mem_conversions_test Random seed: 0x7eace343

==== Test Assertion Failure ====
x86/private_mem_conversions_test.c:46: mem[i] == pattern pid=288745 tid=288747 errno=4 - Interrupted system call
1 0x000000000040426a: memcmp_h at private_mem_conversions_test.c:46
2 (inlined by) __test_mem_conversions at
private_mem_conversions_test.c:356

     3	0x00007f2ea2c89c01: ?? ??:0
     4	0x00007f2ea2d0ec3f: ?? ??:0

Host expected 0xaa at gpa 0x100000000, got 0x0

Quoted 11 lines

Is the ask to also test, under kvm_has_gmem_attributes aka gmem_in_place_conversion, to have guest_memfd used as purely for private memory and have shared memory taken from somewhere else completely?

I can rephrase the changelog and change the code to clarify that this patch is meant to specifically test in-place conversion with MMAP, the intended usage of in-place conversions.

The ask to test gmem for private and something else for shared seems like a different test that can be added separately, perhaps in a follow-up series?

I see it as the fix for existing testcase, not another new test. And the fix is simple:
1. change to use vm_mem_set_memory_attributes() to set memory attribute.
2. pass GUEST_MEMFD_FLAG_INIT_SHARED to create the gmem because this test expects the initial state of the gfn is shared.

---8<----

tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -1
index cf50e9a332c5..fbd6c63d21cf 100644
@@ -307,7 +307,7 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu)
                vm_guest_mem_fallocate(vm, gpa, size, map_shared);
 
        if (set_attributes)
-               vm_set_memory_attributes(vm, gpa, size,
+               vm_mem_set_memory_attributes(vm, gpa, size,
                                         map_shared ? 0 : 
KVM_MEMORY_ATTRIBUTE_PRIVATE);
        run->hypercall.ret = 0;
 }
@@ -382,6 +382,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v
        const size_t slot_size = memfd_size / nr_memslots;
        struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
        pthread_t threads[KVM_MAX_VCPUS];
+       u64 gmem_flags = 0;
        struct kvm_vm *vm;
        int memfd, i;

@@ -397,7 +398,10 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v

        vm_enable_cap(vm, KVM_CAP_EXIT_HYPERCALL, (1 << 

KVM_HC_MAP_GPA_RANGE));

- memfd = vm_create_guest_memfd(vm, memfd_size, 0);
+ if (kvm_has_gmem_attributes)
+ gmem_flags = GUEST_MEMFD_FLAG_INIT_SHARED;
+
+ memfd = vm_create_guest_memfd(vm, memfd_size, gmem_flags);

for (i = 0; i < nr_memslots; i++)
vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i,

#142 lore raw

Re: [PATCH v10 40/41] KVM: selftests: Update private_mem_conversions_test to mmap() guest_memfd

Quoted 60 lines · Xiaoyao Li
Quoted 20 lines · Ackerley Tng
Quoted 18 lines · Sean Christopherson
Quoted 17 lines · Xiaoyao Li
Quoted 11 lines · On 8/8/2026 5:53 AM, Ackerley Tng via B4 Relay wrote:

From: Ackerley Tng<[email protected]>

Update the private memory conversions selftest to also test conversions that are done "in-place" via per-guest_memfd memory attributes. In-place conversions require the host to be able to mmap() the guest_memfd so that the host and guest can share the same backing physical memory.

This includes several updates, that are conditioned on the system supporting per-guest_memfd attributes (kvm_has_gmem_attributes):

1. Set up guest_memfd requesting MMAP and INIT_SHARED.

If I understand correctly, even with in-place gmem, the model that shared memory comes from the userspace_addr and private memory comes from gmem can still work without passing GUEST_MEMFD_FLAG_MMAP flag.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Since this model is not prohibited by KVM. I think we should keep the testcase for this model to ensure test coverage?

+1. Deprecated doesn't mean unsupported.

The original test case was for non-in-place conversion, which wasn't removed, it still passes.

It only passes when

/sys/module/kvm/parameters/gmem_in_place_conversion=n

It fails as below when gmem_in_place_conversion=y

./tools/testing/selftests/kvm/x86/private_mem_conversions_test Random seed: 0x7eace343

==== Test Assertion Failure ====
x86/private_mem_conversions_test.c:46: mem[i] == pattern pid=288745 tid=288747 errno=4 - Interrupted system call
1 0x000000000040426a: memcmp_h at private_mem_conversions_test.c:46
2 (inlined by) __test_mem_conversions at
private_mem_conversions_test.c:356

      3	0x00007f2ea2c89c01: ?? ??:0
      4	0x00007f2ea2d0ec3f: ?? ??:0

Host expected 0xaa at gpa 0x100000000, got 0x0

Quoted 11 lines

Is the ask to also test, under kvm_has_gmem_attributes aka gmem_in_place_conversion, to have guest_memfd used as purely for private memory and have shared memory taken from somewhere else completely?

I can rephrase the changelog and change the code to clarify that this patch is meant to specifically test in-place conversion with MMAP, the intended usage of in-place conversions.

The ask to test gmem for private and something else for shared seems like a different test that can be added separately, perhaps in a follow-up series?

Quoted 6 lines

I see it as the fix for existing testcase, not another new test. And the fix is simple:
1. change to use vm_mem_set_memory_attributes() to set memory attribute.
2. pass GUEST_MEMFD_FLAG_INIT_SHARED to create the gmem because this test expects the initial state of the gfn is shared.

I understand now. I worked it out, I should be able to post v11 tomorrow, just need a little bit more testing.

Quoted 40 lines

---8<----
diff --git a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c index cf50e9a332c5..fbd6c63d21cf 100644

tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -1
@@ -307,7 +307,7 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu)
                 vm_guest_mem_fallocate(vm, gpa, size, map_shared);
 
         if (set_attributes)
-               vm_set_memory_attributes(vm, gpa, size,
+               vm_mem_set_memory_attributes(vm, gpa, size,
                                          map_shared ? 0 :
 KVM_MEMORY_ATTRIBUTE_PRIVATE);
         run->hypercall.ret = 0;
  }
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +0 -0
@@ -382,6 +382,7 @@ static void test_mem_conversions(enum

vm_mem_backing_src_type src_type, u32 nr_v

         const size_t slot_size = memfd_size / nr_memslots;
         struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
         pthread_t threads[KVM_MAX_VCPUS];
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -0
+       u64 gmem_flags = 0;
         struct kvm_vm *vm;
         int memfd, i;
@@ -397,7 +398,10 @@ static void test_mem_conversions(enum

vm_mem_backing_src_type src_type, u32 nr_v

         vm_enable_cap(vm, KVM_CAP_EXIT_HYPERCALL, (1 <<

KVM_HC_MAP_GPA_RANGE));

tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +4 -1
-       memfd = vm_create_guest_memfd(vm, memfd_size, 0);
+       if (kvm_has_gmem_attributes)
+               gmem_flags = GUEST_MEMFD_FLAG_INIT_SHARED;
+
+       memfd = vm_create_guest_memfd(vm, memfd_size, gmem_flags);
 
         for (i = 0; i < nr_memslots; i++)
                 vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i,