Skip to contentSkip to conversation

KVM: guest_memfd: reclaim()/convert() cleanups

PATCH Superseded Sean Christopherson 5 participants 37 messages Jul 14 – Jul 23

Subsystems (4)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v5 cover KVM: guest_memfd: reclaim()/convert() cleanups

These are the cleanups from the SNP VMSA series. I kept it as v5 to maintain the paper trail.

Rework prepare()+invalidate() into convert()+reclaim(), and have the x86 backend use a single convert() flow for both (only SNP needs the hooks, and for SNP, reclaim just means "make shared"). The goals are to eliminate the misleading use of the term "invalidate", to drop a kvm_x86_ops callback, and to prepare for in-place conversion, e.g. so that KVM ends up with something like:

	if (!to_private)
		kvm_gmem_make_shared(inode, start, end);

instead of a funky:

	if (!to_private)
		kvm_gmem_invalidate(inode, start, end);

v5
- Drop the VMSA fixes (already applied).
- Keep a dedicated reclaim() flow. [Fuad, Ackerley]
- Pass a gfn, not a gpa, to convert() / reclaim(). [Ackerley]
- Collect reviews. [Ackerley]
- Align the gfn when folding __kvm_gmem_prepare_folio() into its caller. [Sashiko]

v4:
- https://lore.kernel.org/all/[email protected]

- Collect reviews. [Mike, Ackerley]
- Rework prepare()+invalidate() into a common convert() instead of simply renaming invalidate() to reclaim_memory(). [Ackerley]
- Fix several (mostly benign) issues in the "prepare" flow.

v3:
- https://lore.kernel.org/all/[email protected]
- Ensure disabling quirks while the VM is live won't result in KVM skipping the back-half of "zap all fast". [Sashiko]
- s/gmem_free_folio/gmem_reclaim_memory. [Ackerley]
- Use {READ,WRITE}_ONCE() when accessing the guest's VMSA GPA outside of the per-vCPU mutex, and comment. [Sashiko]

v2:
- https://lore.kernel.org/all/[email protected]
- Invalidate VMSAs if the memslot is DELETED or MOVED. [Sashiko]
- Limit stable@ patches without a Fixes to 6.12+.

v1: https://lore.kernel.org/all/[email protected]

Sean Christopherson (7):
KVM: guest_memfd: Plumb the number of pages and max order into
.invalidate()
KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and
isolate it
KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare()
failure
KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
KVM: guest_memfd: Explicitly pass number of pages to
kvm_arch_gmem_prepare()
KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into

    .gmem_convert()

KVM: guest_memfd: Rework PREPARE config and hook into a more generic
CONVERT

 arch/x86/include/asm/kvm-x86-ops.h |  5 ++--
 arch/x86/include/asm/kvm_host.h    |  6 ++--
 arch/x86/kvm/Kconfig               |  3 +-
 arch/x86/kvm/svm/sev.c             | 23 +++++++++++++--
 arch/x86/kvm/svm/svm.c             |  3 +-
 arch/x86/kvm/svm/svm.h             |  4 +--
 arch/x86/kvm/x86.c                 | 18 +++++++-----
 include/linux/kvm_host.h           | 10 +++++--
 virt/kvm/Kconfig                   |  6 +++-
 virt/kvm/guest_memfd.c             | 46 +++++++++++-------------------
 10 files changed, 70 insertions(+), 54 deletions(-)

base-commit: 6bc96b971766fbbbbdd9fb2642cedacaf02da957

Signature · 1 line
2.55.0.141.g00534a21ce-goog
author maintainer
#1 lore raw

PATCH v5 1/7 KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

Plumb the number of pages to "invalidate", i.e. reclaim, along with the max order with which the pages can be mapped. This will allow x86 to combine its vendor-specific implementations for conversion and reclaim, a.k.a. preparation and invalidation, into a single hook.

No functional change intended.

---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 2 +-
 virt/kvm/guest_memfd.c   | 7 ++-----
 3 files changed, 5 insertions(+), 8 deletions(-)
arch/x86/kvm/x86.c +2 -2
index 1f5dc685f049..a2cbf6fae118 100644
@@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-	kvm_x86_call(gmem_invalidate)(start, end);
+	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
include/linux/kvm_host.h +1 -1
index 9db6eb4023c4..996b39ebdb68 100644
@@ -2614,7 +2614,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
virt/kvm/guest_memfd.c +2 -5
index 171a9da1b685..99c070bfd446 100644
@@ -530,11 +530,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-	struct page *page = folio_page(folio, 0);
-	kvm_pfn_t pfn = page_to_pfn(page);
-	int order = folio_order(folio);
-
-	kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
+	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+				 folio_order(folio));
 }
 #endif
 
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Reviewed-by on v5 1/7
#2 lore raw

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 533

Quoted 12 lines · Sean Christopherson

[...snip...]

#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE static void kvm_gmem_free_folio(struct folio *folio) {

virt/kvm/guest_memfd.c +1 -5
-	struct page *page = folio_page(folio, 0);
-	kvm_pfn_t pfn = page_to_pfn(page);
-	int order = folio_order(folio);
-
-	kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
+	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),

How about just folio_pfn(folio)? Since we're freeing the folio this will always be 0.

Quoted 4 lines
virt/kvm/guest_memfd.c +1 -0
+				 folio_order(folio));
 }
 #endif
Reviewed-by on v5 1/7
#3 lore raw

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

In reply to Sean Christopherson
Quoted 36 lines · Sean Christopherson

Plumb the number of pages to "invalidate", i.e. reclaim, along with the max order with which the pages can be mapped. This will allow x86 to combine its vendor-specific implementations for conversion and reclaim, a.k.a. preparation and invalidation, into a single hook.

No functional change intended.

---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 2 +-
 virt/kvm/guest_memfd.c   | 7 ++-----
 3 files changed, 5 insertions(+), 8 deletions(-)
arch/x86/kvm/x86.c +2 -2
index 1f5dc685f049..a2cbf6fae118 100644
@@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-       kvm_x86_call(gmem_invalidate)(start, end);
+       kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
include/linux/kvm_host.h +0 -0
index 9db6eb4023c4..996b39ebdb68 100644

include/linux/kvm_host.h long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, · line 2617

Quoted 5 lines
include/linux/kvm_host.h +1 -1
@@ -2614,7 +2614,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);

nit: the declaration names the last parameter "order" here, but the definition in arch/x86/kvm/x86.c calls it "max_order".

Cheers,
/fuad

Quoted 24 lines
include/linux/kvm_host.h +0 -0
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
virt/kvm/guest_memfd.c +2 -6
index 171a9da1b685..99c070bfd446 100644
@@ -530,11 +530,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-       struct page *page = folio_page(folio, 0);
-       kvm_pfn_t pfn = page_to_pfn(page);
-       int order = folio_order(folio);
-
-       kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
+       kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+                                folio_order(folio));
 }
 #endif
 
--

2.55.0.141.g00534a21ce-goog

#4 lore raw

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

In reply to Sean Christopherson
Quoted 19 lines · Sean Christopherson

Plumb the number of pages to "invalidate", i.e. reclaim, along with the max order with which the pages can be mapped. This will allow x86 to combine its vendor-specific implementations for conversion and reclaim, a.k.a. preparation and invalidation, into a single hook.

No functional change intended.

---
  arch/x86/kvm/x86.c       | 4 ++--
  include/linux/kvm_host.h | 2 +-
  virt/kvm/guest_memfd.c   | 7 ++-----
  3 files changed, 5 insertions(+), 8 deletions(-)
arch/x86/kvm/x86.c +0 -0
index 1f5dc685f049..a2cbf6fae118 100644

arch/x86/kvm/x86.c int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord · line 10630

Quoted 5 lines
arch/x86/kvm/x86.c +1 -1
@@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
  #endif
 
  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)

why use 'kvm_pfn_t' as the type for nr_pages instead of 'unsigned long'?

(I guess just because you kept the original type for @end)

author maintainer
#5 lore raw

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

Quoted 27 lines · Xiaoyao Li
Quoted 19 lines · Sean Christopherson

Plumb the number of pages to "invalidate", i.e. reclaim, along with the max order with which the pages can be mapped. This will allow x86 to combine its vendor-specific implementations for conversion and reclaim, a.k.a. preparation and invalidation, into a single hook.

No functional change intended.

---
  arch/x86/kvm/x86.c       | 4 ++--
  include/linux/kvm_host.h | 2 +-
  virt/kvm/guest_memfd.c   | 7 ++-----
  3 files changed, 5 insertions(+), 8 deletions(-)
arch/x86/kvm/x86.c +0 -0
index 1f5dc685f049..a2cbf6fae118 100644

arch/x86/kvm/x86.c int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord · line 10629

Quoted 4 lines
arch/x86/kvm/x86.c +1 -1
@@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
  #endif
  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)

why use 'kvm_pfn_t' as the type for nr_pages instead of 'unsigned long'?

Quoted 1 line

(I guess just because you kept the original type for @end)

Because the number of pages, i.e. number of PFNs, need to be in the same "domain" as the staring PFN (and KVM's MMU needs to treat PFNs as u64s).

author maintainer
#6 lore raw

PATCH v5 2/7 KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

In reply to Sean Christopherson

Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its own RECLAIM Kconfig, as the hook is called when a folio is freed, which is far too late and lacks sufficient information for KVM to actually invalidate its usage of the memory. E.g. SNP uses the hook to convert memory back to SHARED so that it can be safely accessed by the host, there is no invalidation of guest mappings anywhere. Isolating the hook will also allow pKVM on arm64 to opt-in to reclaim() without also having to differentiate between reclaim and conversions to shared for active VMs.

Keep guest_memfd's trampoline, even though it would be trivial to wire up .free_folio() directly to an arch callback, to avoid bleeding guest_memfd internals into arch code (specifically, avoid referencing folios in arch code).

Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped entirely ("reclaim" on SNP is the same as convert-to-shared).

---
 arch/x86/kvm/Kconfig     | 1 +
 arch/x86/kvm/x86.c       | 7 +++++--
 include/linux/kvm_host.h | 5 ++++-
 virt/kvm/Kconfig         | 4 ++++
 virt/kvm/guest_memfd.c   | 8 ++++----
 5 files changed, 18 insertions(+), 7 deletions(-)
arch/x86/kvm/Kconfig +1 -0
index 801bf9e520db..e0e7ad015839 100644
@@ -161,6 +161,7 @@ config KVM_AMD_SEV
 	select ARCH_HAS_CC_PLATFORM
 	select KVM_GENERIC_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
arch/x86/kvm/x86.c +5 -2
index a2cbf6fae118..08d24c6fffa4 100644
@@ -10626,11 +10626,14 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 }
 #endif
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
 	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
+#endif
+
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_x86_call(gmem_invalidate_range)(kvm, range);
include/linux/kvm_host.h +4 -1
index 996b39ebdb68..ee87ddd3294f 100644
@@ -2613,8 +2613,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 		       kvm_gmem_populate_cb post_populate, void *opaque);
 #endif
 
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
+#endif
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
virt/kvm/Kconfig +4 -0
index 794976b88c6f..617876993225 100644
@@ -111,6 +111,10 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
        bool
        depends on KVM_GUEST_MEMFD
 
+config HAVE_KVM_ARCH_GMEM_RECLAIM
+       bool
+       depends on KVM_GUEST_MEMFD
+
 config HAVE_KVM_ARCH_GMEM_INVALIDATE
        bool
        depends on KVM_GUEST_MEMFD
virt/kvm/guest_memfd.c +4 -4
index 99c070bfd446..d5d227476cda 100644
@@ -527,11 +527,11 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	return MF_DELAYED;
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
-				 folio_order(folio));
+	kvm_arch_gmem_reclaim(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+			      folio_order(folio));
 }
 #endif
 
@@ -539,7 +539,7 @@ static const struct address_space_operations kvm_gmem_aops = {
 	.dirty_folio = noop_dirty_folio,
 	.migrate_folio	= kvm_gmem_migrate_folio,
 	.error_remove_folio = kvm_gmem_error_folio,
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 	.free_folio = kvm_gmem_free_folio,
 #endif
 };
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Reviewed-by on v5 2/7
#7 lore raw

Re: [PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

Quoted 14 lines · Sean Christopherson

Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its own RECLAIM Kconfig, as the hook is called when a folio is freed, which is far too late and lacks sufficient information for KVM to actually invalidate its usage of the memory. E.g. SNP uses the hook to convert memory back to SHARED so that it can be safely accessed by the host, there is no invalidation of guest mappings anywhere. Isolating the hook will also allow pKVM on arm64 to opt-in to reclaim() without also having to differentiate between reclaim and conversions to shared for active VMs.

Keep guest_memfd's trampoline, even though it would be trivial to wire up .free_folio() directly to an arch callback, to avoid bleeding guest_memfd internals into arch code (specifically, avoid referencing folios in arch code).

Quoted 6 lines

Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped entirely ("reclaim" on SNP is the same as convert-to-shared).

[...snip...]

Reviewed-by on v5 2/7
#8 lore raw

Re: [PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

In reply to Sean Christopherson
Quoted 15 lines · Sean Christopherson

Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its own RECLAIM Kconfig, as the hook is called when a folio is freed, which is far too late and lacks sufficient information for KVM to actually invalidate its usage of the memory. E.g. SNP uses the hook to convert memory back to SHARED so that it can be safely accessed by the host, there is no invalidation of guest mappings anywhere. Isolating the hook will also allow pKVM on arm64 to opt-in to reclaim() without also having to differentiate between reclaim and conversions to shared for active VMs.

Keep guest_memfd's trampoline, even though it would be trivial to wire up .free_folio() directly to an arch callback, to avoid bleeding guest_memfd internals into arch code (specifically, avoid referencing folios in arch code).

Quoted 6 lines

Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped entirely ("reclaim" on SNP is the same as convert-to-shared).

Thanks Sean, this matches what pKVM needs: we'll select HAVE_KVM_ARCH_GMEM_RECLAIM and implement kvm_arch_gmem_reclaim() only, without the convert hook, so teardown (which for pKVM must scrub the page before it returns to the host) stays distinct from a private->shared conversion (which must preserve it).

Same nit as patch 1 (order/max_order).

Cheers,
/fuad

Quoted 104 lines
---
 arch/x86/kvm/Kconfig     | 1 +
 arch/x86/kvm/x86.c       | 7 +++++--
 include/linux/kvm_host.h | 5 ++++-
 virt/kvm/Kconfig         | 4 ++++
 virt/kvm/guest_memfd.c   | 8 ++++----
 5 files changed, 18 insertions(+), 7 deletions(-)
arch/x86/kvm/Kconfig +1 -0
index 801bf9e520db..e0e7ad015839 100644
@@ -161,6 +161,7 @@ config KVM_AMD_SEV
        select ARCH_HAS_CC_PLATFORM
        select KVM_GENERIC_MEMORY_ATTRIBUTES
        select HAVE_KVM_ARCH_GMEM_PREPARE
+       select HAVE_KVM_ARCH_GMEM_RECLAIM
        select HAVE_KVM_ARCH_GMEM_INVALIDATE
        select HAVE_KVM_ARCH_GMEM_POPULATE
        help
arch/x86/kvm/x86.c +5 -2
index a2cbf6fae118..08d24c6fffa4 100644
@@ -10626,11 +10626,14 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 }
 #endif
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
        kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
+#endif
+
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        kvm_x86_call(gmem_invalidate_range)(kvm, range);
include/linux/kvm_host.h +4 -1
index 996b39ebdb68..ee87ddd3294f 100644
@@ -2613,8 +2613,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
                       kvm_gmem_populate_cb post_populate, void *opaque);
 #endif
 
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
+#endif
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
virt/kvm/Kconfig +4 -0
index 794976b88c6f..617876993225 100644
@@ -111,6 +111,10 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
        bool
        depends on KVM_GUEST_MEMFD
 
+config HAVE_KVM_ARCH_GMEM_RECLAIM
+       bool
+       depends on KVM_GUEST_MEMFD
+
 config HAVE_KVM_ARCH_GMEM_INVALIDATE
        bool
        depends on KVM_GUEST_MEMFD
virt/kvm/guest_memfd.c +4 -5
index 99c070bfd446..d5d227476cda 100644
@@ -527,11 +527,11 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
        return MF_DELAYED;
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-       kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
-                                folio_order(folio));
+       kvm_arch_gmem_reclaim(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+                             folio_order(folio));
 }
 #endif
 
@@ -539,7 +539,7 @@ static const struct address_space_operations kvm_gmem_aops = {
        .dirty_folio = noop_dirty_folio,
        .migrate_folio  = kvm_gmem_migrate_folio,
        .error_remove_folio = kvm_gmem_error_folio,
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
        .free_folio = kvm_gmem_free_folio,
 #endif
 };
--

2.55.0.141.g00534a21ce-goog

Reviewed-by on v5 2/7
#9 lore raw

Re: [PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

In reply to Sean Christopherson
Quoted 45 lines · Sean Christopherson

Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its own RECLAIM Kconfig, as the hook is called when a folio is freed, which is far too late and lacks sufficient information for KVM to actually invalidate its usage of the memory. E.g. SNP uses the hook to convert memory back to SHARED so that it can be safely accessed by the host, there is no invalidation of guest mappings anywhere. Isolating the hook will also allow pKVM on arm64 to opt-in to reclaim() without also having to differentiate between reclaim and conversions to shared for active VMs.

Keep guest_memfd's trampoline, even though it would be trivial to wire up .free_folio() directly to an arch callback, to avoid bleeding guest_memfd internals into arch code (specifically, avoid referencing folios in arch code).

Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped entirely ("reclaim" on SNP is the same as convert-to-shared).

---
  arch/x86/kvm/Kconfig     | 1 +
  arch/x86/kvm/x86.c       | 7 +++++--
  include/linux/kvm_host.h | 5 ++++-
  virt/kvm/Kconfig         | 4 ++++
  virt/kvm/guest_memfd.c   | 8 ++++----
  5 files changed, 18 insertions(+), 7 deletions(-)
arch/x86/kvm/Kconfig +1 -0
index 801bf9e520db..e0e7ad015839 100644
@@ -161,6 +161,7 @@ config KVM_AMD_SEV
  	select ARCH_HAS_CC_PLATFORM
  	select KVM_GENERIC_MEMORY_ATTRIBUTES
  	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_RECLAIM
  	select HAVE_KVM_ARCH_GMEM_INVALIDATE
  	select HAVE_KVM_ARCH_GMEM_POPULATE
  	help
arch/x86/kvm/x86.c +0 -0
index a2cbf6fae118..08d24c6fffa4 100644

arch/x86/kvm/x86.c int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord · line 10630

Quoted 7 lines
arch/x86/kvm/x86.c +2 -2
@@ -10626,11 +10626,14 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
  }
  #endif
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)

The kvm_pfn_t for nr_pages still catches my eye, but it's not the business of this patch.

author maintainer
#10 lore raw

PATCH v5 3/7 KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

In reply to Sean Christopherson

Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP alreads logs more precise messages in all error paths, and whether or not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail.

---
 virt/kvm/guest_memfd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +3 -7
index d5d227476cda..62c2c157f0c1 100644
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	kvm_pfn_t pfn = folio_file_pfn(folio, index);
 	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-	int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-	if (rc) {
-		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
-				    index, gfn, pfn, rc);
-		return rc;
-	}
-#endif
 
+	return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+#else
 	return 0;
+#endif
 }
 
 /*
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Reviewed-by on v5 3/7
#11 lore raw

Re: [PATCH v5 3/7] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

Quoted 3 lines · Sean Christopherson

Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP alreads logs more precise messages in all error paths, and whether or

nit: already

Cheers,
/fuad

Quoted 35 lines

not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail.

---
 virt/kvm/guest_memfd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +3 -8
index d5d227476cda..62c2c157f0c1 100644
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
        kvm_pfn_t pfn = folio_file_pfn(folio, index);
        gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-       int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-       if (rc) {
-               pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
-                                   index, gfn, pfn, rc);
-               return rc;
-       }
-#endif
 
+       return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+#else
        return 0;
+#endif
 }
 
 /*
--

2.55.0.141.g00534a21ce-goog

Reviewed-by on v5 3/7
#12 lore raw

Re: [PATCH v5 3/7] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

In reply to Sean Christopherson
Quoted 7 lines · Sean Christopherson

Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP alreads logs more precise messages in all error paths, and whether or not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail.

Quoted 27 lines
---
  virt/kvm/guest_memfd.c | 10 +++-------
  1 file changed, 3 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +3 -7
index d5d227476cda..62c2c157f0c1 100644
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
  	kvm_pfn_t pfn = folio_file_pfn(folio, index);
  	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-	int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-	if (rc) {
-		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
-				    index, gfn, pfn, rc);
-		return rc;
-	}
-#endif
 
+	return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+#else
  	return 0;
+#endif
  }
 
  /*
author maintainer
#13 lore raw

PATCH v5 4/7 KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

In reply to Sean Christopherson

Fold __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio() and extend the #ifdef guard to cover all of the kvm_gmem_prepare_folio(). Take care to align the gfn _before_ computing the index, otherwise KVM would end up with a mismatched gfn+pfn pair for folios larger than order-0 (this detail was subtly handled by recomputing gfn in the helper using the aligned pfn).

Providing a single-use, ~2 line tail-call helper adds no value. In fact, due to having to reconstitute the gfn, *and* effectively hiding the alignment adjustment for hugepages, the separate helper is very arguably a net negative.

Opportunistically convert the existing WARN in the prepare flow to a only fire once, e.g. so that a guest_memfd bug doesn't unintentionally Dos the kernel by spamming the log.

No functional change intended.

---
 virt/kvm/guest_memfd.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)
virt/kvm/guest_memfd.c +8 -16
index 62c2c157f0c1..f9a1d6d64f6c 100644
@@ -60,19 +60,6 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
 	return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
-static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				    pgoff_t index, struct folio *folio)
-{
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	kvm_pfn_t pfn = folio_file_pfn(folio, index);
-	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-
-	return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-#else
-	return 0;
-#endif
-}
-
 /*
  * Process @folio, which contains @gfn, so that the guest can use it.
  * The folio must be locked and the gfn must be contained in @slot.
@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	pgoff_t index;
 
 	/*
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
+				     folio_order(folio));
+#else
+	return 0;
+#endif
 }
 
 /*
Signature · 1 line
2.55.0.141.g00534a21ce-goog
#14 lore raw

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Refactoring kvm_gmem_prepare_folio() and kvm_gmem_make_private() to align gfn before computing index produces a mismatched and unaligned pfn if the memslot's base_gfn is not aligned to the huge folio size. This causes the architecture-specific prepare function to write beyond the bounds of the folio, leading to host physical memory corruption.

Pre-existing issues:
- [High] In SEV-SNP, if large RMP entry preparation fails for a huge folio, KVM will only prepare the base 4K page, causing an incomplete initialization and an infinite page fault loop for subpage accesses. --

--- Patch [4]: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller --- commit ef3a2643968d189a00f8b26552b9b814bf1aa548
Author: Sean Christopherson <[email protected]>

KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

This commit folds __kvm_gmem_prepare_folio() into its caller, removing the single-use tail-call helper. It also adjusts the alignment logic to align the gfn before computing the index, and converts a WARN to WARN_ON_ONCE.

Quoted 4 lines
virt/kvm/guest_memfd.c +0 -0
index 28cdcf6007453..f065dedfbba8c 100644

[ ... ]

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 93

Quoted 13 lines
virt/kvm/guest_memfd.c +4 -3
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
+				     folio_order(folio));

[Severity: Critical]
Does aligning gfn before computing index cause an unaligned index when slot->base_gfn is not huge-page aligned?

If a userspace process maps a guest_memfd using KVM_SET_USER_MEMORY_REGION2 with a base_gfn that is not aligned to the huge page size, kvm_gmem_get_index() could return an unaligned index.

Because folio_file_pfn() applies the index offset to the base PFN:

virt/kvm/guest_memfd.c:folio_file_pfn() {
    return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
}

an unaligned index causes it to return an offset PFN instead of the folio's base PFN.

Could this lead to host physical memory corruption? The architecture backend processes the full page count starting from the offset PFN, which would write beyond the allocated folio boundaries and overwrite unrelated host physical memory.

It looks like this same alignment pattern is also present in kvm_gmem_make_private().

[Severity: High]
This isn't a bug introduced by this patch, but could this generic function incorrectly assume a successful return means the entire huge folio was prepared?

Looking at sev_gmem_prepare() in arch/x86/kvm/svm/sev.c, if it falls back to PG_LEVEL_4K for a huge folio:

    } else {
        level = PG_LEVEL_4K;
        pfn_aligned = pfn;
        gfn_aligned = gfn;
    }

    rc = rmp_make_private(pfn_aligned, gfn_to_gpa(gfn_aligned), level, sev->asid, false);

It appears rmp_make_private() is called only once for the base 4K page, without looping over the remaining pages in the huge folio.

Since kvm_arch_gmem_prepare() then returns success to this generic code, are the remaining 511 pages left uninitialized, potentially causing an infinite page fault loop for subsequent subpage accesses?

Quoted 4 lines
virt/kvm/guest_memfd.c +3 -0
+#else
+	return 0;
+#endif
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#15 lore raw

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

In reply to Sean Christopherson
Quoted 5 lines · Sean Christopherson

[...snip...]

virt/kvm/guest_memfd.c +0 -0
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 92

Quoted 10 lines
virt/kvm/guest_memfd.c +3 -3
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),

Could this just be folio_pfn(folio) since this function is kvm_gmem_prepare_folio() and guest_memfd will always try to prepare the entire folio?

Quoted 5 lines
virt/kvm/guest_memfd.c +4 -0
+				     folio_order(folio));
+#else
+	return 0;
+#endif
 }
author maintainer
#16 lore raw

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

Quoted 15 lines · Ackerley Tng
Quoted 13 lines · Sean Christopherson

[...snip...]

virt/kvm/guest_memfd.c +2 -2
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
Quoted 6 lines

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 92

Quoted 2 lines
virt/kvm/guest_memfd.c +1 -1
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),

Could this just be folio_pfn(folio) since this function is kvm_gmem_prepare_folio() and guest_memfd will always try to prepare the entire folio?

No? Maybe? For this patch, I'm just trying to shuffle code araound. Even for this series, I'd prefer not to make any more semantic changes than are needed to get to a sane state, and to prepare for in-place conversion. If there's a need and/or a good reason to use folio_pfn(), by all means, send a patch.

author maintainer
#17 lore raw

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

+Mike

Quoted 23 lines · Sean Christopherson
Quoted 15 lines · Ackerley Tng
Quoted 13 lines · Sean Christopherson

[...snip...]

virt/kvm/guest_memfd.c +2 -2
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
Quoted 6 lines

virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 92

Quoted 2 lines
virt/kvm/guest_memfd.c +1 -1
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),

Could this just be folio_pfn(folio) since this function is kvm_gmem_prepare_folio() and guest_memfd will always try to prepare the entire folio?

Quoted 4 lines

No? Maybe? For this patch, I'm just trying to shuffle code araound. Even for this series, I'd prefer not to make any more semantic changes than are needed to get to a sane state, and to prepare for in-place conversion. If there's a need and/or a good reason to use folio_pfn(), by all means, send a patch.

I take that back. After far too much staring and digging into the history (and future) of this code, I finally see what you were getting at. Because KVM "prepares" entire folios, grabbing a specific page in the folio is nonsensical, just grab the base pfn and be done with it.

However, I'm not convinced to committing harder to per-folio "preparation" is the way to go. The pre-folio prepation was added when guest_memfd actually tracked preparedness. At the time, it made sense, because tracking per-folio meant we didn't need to add a separate data structure to track that information, and doing per-folio tracking only works if the entire folio is prepared (or not).

Now that we dropped that code in favor having SNP query the RMP, per-folio preparation, i.e. per-folio conversions to private, doesn't make as much sense. There is still technically an argument for per-folio conversion, *if* what I described below is even allowed by SNP.

If SNP allows the RMP size to be greater than the NPT size, then per-folio conversion would allow KVM to assign a hugepage in the RMP even if it can only be mapped into the NPT with a smaller page, e.g. because of memslot alignment. The documentation of that reasoning would be something like this:

/*
* If the memory is private from KVM's perspective, and hardware tracks
* VM-assigned private memory in a dedicated data structure, i.e. not
* in the stage-2 page tables, then call into arch code to assign the
* entire folio to the guest. Assigning the entire folio, e.g. instead
* of only the memory being mapped into the guest, allows KVM to assign
* an entire hugepage of memory in the out-of-band structure even if
* KVM can only map a smaller page size into the MMU, e.g. because the
* gmem hugepage is spread across multiple memslots.
*/
if (kvm_gmem_is_private_mem(file_inode(file), index))
r = kvm_gmem_make_private(kvm, slot, gfn, folio);

But it's not clear to me that SNP even supports mismatched page sizes, because several of the flows in the APM explicitly state that triggers an #NPF, and this comment in KVM's sev_handle_rmp_fault() also suggests that's not allow.

* 2) Guest access via NPT can trigger an #NPF if the NPT mapping is
* smaller than what is indicated by the 2MB RMP entry for the PFN
* that backs the GPA.

And even *if* that's allowed by SNP, I'm not at all convinced it's worth doing in KVM, because it should be a rare situation. E.g. maybe for memory at the top of TOLUD that has holes for non-RAM crud or something?

So instead of using folio_pfn(), I'm leaning strongly towards entirely dropping what is currently kvm_gmem_prepare_folio(), and instead converting exactly what is being mapped into the guest. Because I've messed up the gfn and index handling twice now, I'm still struggling to wrap my head around the math, *and* I really don't want to have two separate flows computing "how much memory can be assigned and at what size" once hugepage support comes along.

I.e. do this:

	pgoff_t index = kvm_gmem_get_index(slot, gfn);
	struct folio *folio;
	int r = 0, __order;

	max_order = max_order ?: &__order;

	CLASS(gmem_get_file, file)(slot);
	if (!file)
		return -EFAULT;

	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
	if (IS_ERR(folio))
		return PTR_ERR(folio);

	if (!folio_test_uptodate(folio)) {
		clear_highpage(folio_page(folio, 0));
		folio_mark_uptodate(folio);
	}

	if (kvm_gmem_is_private_mem(file_inode(file), index))
		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn, *max_order);

Am I missing something? Anyone feel strongly about keeping per-folio converions for when SNP supports hugepages?

author maintainer
#18 lore raw

PATCH v5 5/7 KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

In reply to Sean Christopherson

Tweak the guest_memfd prepare() hook to explicitly pass the number of pages to "prepare", so that the prepare() and invalidate() hooks can be combined into a single convert() hook, and because the existing code is outright broken if a guest_memfd folio is comprised of more than one page (which can't happen, yet). guest_memfd aligns the pfn down, and the SNP code *tries* to create a corresponding huge entry, but if the RMP must use 4KiB entries for whatever reason, KVM will only "prepare" the first pfn, and not the actual pfn that needs to be converted/prepared.

Alternatively, @max_order could simply be repurposed as _the_ @order, but that will fall apart when in-place conversion comes along, at which point KVM will need to deal with conversions that aren't strictly bound to a one folio.

WARN in the SNP code if the number of pages to prepare is anything other than '1', i.e. if guest_memfd is trying to prepare/convert more than a single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion greater than order-0 folios.

Opportunistically swap the ordering of @pfn and @gfn params for kvm_x86_ops.gmem_prepare() to match that of kvm_arch_gmem_prepare().

Fixes b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
---
 arch/x86/include/asm/kvm_host.h | 3 ++-
 arch/x86/kvm/svm/sev.c          | 6 +++++-
 arch/x86/kvm/svm/svm.h          | 3 ++-
 arch/x86/kvm/x86.c              | 5 +++--
 include/linux/kvm_host.h        | 3 ++-
 virt/kvm/guest_memfd.c          | 7 ++++---
 6 files changed, 18 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
index 58f156ae31e7..e7df537d36d5 100644
@@ -1725,7 +1725,8 @@ struct kvm_x86_ops {
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+	int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order);
 #endif
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/svm/sev.c +5 -1
index 82b983484af2..adc58138fdd0 100644
@@ -5114,7 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
+int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	kvm_pfn_t pfn_aligned;
@@ -5125,6 +5126,9 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
 	if (!sev_snp_guest(kvm))
 		return 0;
 
+	if (WARN_ON_ONCE(nr_pages != 1))
+		return -EIO;
+
 	rc = snp_lookup_rmpentry(pfn, &assigned, &level);
 	if (rc) {
 		pr_err_ratelimited("SEV: Failed to look up RMP entry: GFN %llx PFN %llx error %d\n",
arch/x86/kvm/svm/svm.h +2 -1
index af25e4b56212..e35419456648 100644
@@ -1009,7 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order);
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
arch/x86/kvm/x86.c +3 -2
index 08d24c6fffa4..d484d8b509b1 100644
@@ -10620,9 +10620,10 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
+int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, pfn, gfn, max_order);
+	return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
 }
 #endif
 
include/linux/kvm_host.h +2 -1
index ee87ddd3294f..0a06234a8a8b 100644
@@ -2580,7 +2580,8 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
+int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +4 -3
index f9a1d6d64f6c..0f6745f3b919 100644
@@ -70,6 +70,7 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
 	/*
@@ -85,12 +86,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
-	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
+	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
 
 	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     folio_order(folio));
+				     nr_pages, folio_order(folio));
 #else
 	return 0;
 #endif
Signature · 1 line
2.55.0.141.g00534a21ce-goog
author maintainer
#19 lore raw

PATCH v5 6/7 KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

In reply to Sean Christopherson

Smush x86's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin for SNP: they're invoked when private/shared memory is about to made accessible/visible to the guest or host. I.e. prepare() is really "make private", and invalidate() is really "make shared".

Using a single hook will yield more intuitive code when in-place conversion comes along.

For all intents and purposes, no functional change intended.

---
 arch/x86/include/asm/kvm-x86-ops.h |  5 ++---
 arch/x86/include/asm/kvm_host.h    |  7 +++----
 arch/x86/kvm/svm/sev.c             | 21 ++++++++++++++++++---
 arch/x86/kvm/svm/svm.c             |  3 +--
 arch/x86/kvm/svm/svm.h             |  5 ++---
 arch/x86/kvm/x86.c                 |  4 ++--
 6 files changed, 28 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -3
index 736129db272a..76ce691de2f8 100644
@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
-KVM_X86_OP_OPTIONAL(gmem_invalidate)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
 #endif
arch/x86/include/asm/kvm_host.h +3 -4
index e7df537d36d5..f60fec14c61a 100644
@@ -1724,11 +1724,10 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			    kvm_pfn_t nr_pages, int max_order);
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
-	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 	void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
arch/x86/kvm/svm/sev.c +18 -3
index adc58138fdd0..b00623b2f26a 100644
@@ -5114,8 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order)
+static int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+				 kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	kvm_pfn_t pfn_aligned;
@@ -5165,8 +5165,9 @@ int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	return 0;
 }
 
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+static void sev_gmem_make_shared(kvm_pfn_t start, kvm_pfn_t nr_pages)
 {
+	kvm_pfn_t end = start + nr_pages;
 	kvm_pfn_t pfn;
 
 	if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
@@ -5228,6 +5229,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 	}
 }
 
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int order, bool to_private)
+{
+	if (to_private) {
+		if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))
+			return -EIO;
+
+		return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
+	}
+
+	sev_gmem_make_shared(pfn, nr_pages);
+	return 0;
+}
+
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	struct kvm_vcpu *vcpu;
arch/x86/kvm/svm/svm.c +1 -2
index 91286d46d13a..8c5018c65dc5 100644
@@ -5438,8 +5438,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
 	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
 
-	.gmem_prepare = sev_gmem_prepare,
-	.gmem_invalidate = sev_gmem_invalidate,
+	.gmem_convert = sev_gmem_convert,
 	.gmem_invalidate_range = sev_gmem_invalidate_range,
 	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
arch/x86/kvm/svm/svm.h +2 -3
index e35419456648..57f692b5d709 100644
@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order);
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order, bool to_private);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +2 -2
index d484d8b509b1..e509d50c247c 100644
@@ -10623,14 +10623,14 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			  kvm_pfn_t nr_pages, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
+	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
 }
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
 }
 #endif
 
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Reviewed-by on v5 6/7
#20 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

arch/x86/kvm/svm/sev.c void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) · line 5236

Quoted 5 lines · Sean Christopherson
arch/x86/kvm/svm/sev.c +5 -0
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int order, bool to_private)
+{
+	if (to_private) {
+		if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))

Omg, maxphyaddr is actually the number of bits in an address and not an address!

Thanks for explaining on v4.

Quoted 8 lines
arch/x86/kvm/svm/sev.c +8 -0
+			return -EIO;
+
+		return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
+	}
+
+	sev_gmem_make_shared(pfn, nr_pages);
+	return 0;
+}

arch/x86/kvm/svm/sev.c void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)

Quoted 10 lines
arch/x86/kvm/svm/sev.c +2 -1
+
 
[...snip...]
 
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));

This setup means later pKVM would have to define a .gmem_convert but actually only use it for reclaim, which seems kind of an odd definition. (We can fix this later.)

Quoted 5 lines

}
#endif

--
2.55.0.141.g00534a21ce-goog

author maintainer
#21 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

Quoted 10 lines · Ackerley Tng

arch/x86/kvm/x86.c bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm) · line 10633

Quoted 5 lines · Sean Christopherson

#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) {

arch/x86/kvm/x86.c +1 -1
-	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));

This setup means later pKVM would have to define a .gmem_convert but actually only use it for reclaim, which seems kind of an odd definition. (We can fix this later.)

No, pKVM (on arm64) would just implement kvm_arch_gmem_reclaim(), and then do whatever it wants under the hood.

#22 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

Quoted 9 lines · Sean Christopherson
Quoted 7 lines · Ackerley Tng
Quoted 5 lines · Sean Christopherson

#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) {

arch/x86/kvm/x86.c +1 -1
-   kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+   WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
Quoted 6 lines
Quoted 3 lines

This setup means later pKVM would have to define a .gmem_convert but actually only use it for reclaim, which seems kind of an odd definition. (We can fix this later.)

No, pKVM (on arm64) would just implement kvm_arch_gmem_reclaim(), and then do whatever it wants under the hood.

Right. pKVM would just select HAVE_KVM_ARCH_GMEM_RECLAIM and implement kvm_arch_gmem_reclaim() to scrub; the reclaim and convert hooks have independent Kconfigs.

Thank you Ackerley for checking, and thank you all for the respin.

Cheers,
/fuad

Reviewed-by on v5 6/7
#23 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

In reply to Sean Christopherson
Quoted 7 lines · Sean Christopherson

Smush x86's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin for SNP: they're invoked when private/shared memory is about to made accessible/visible to the guest or host. I.e. prepare() is really "make private", and invalidate() is really "make shared".

Quoted 6 lines

Using a single hook will yield more intuitive code when in-place conversion comes along.

For all intents and purposes, no functional change intended.

Cheers,
/fuad

Quoted 146 lines
---
 arch/x86/include/asm/kvm-x86-ops.h |  5 ++---
 arch/x86/include/asm/kvm_host.h    |  7 +++----
 arch/x86/kvm/svm/sev.c             | 21 ++++++++++++++++++---
 arch/x86/kvm/svm/svm.c             |  3 +--
 arch/x86/kvm/svm/svm.h             |  5 ++---
 arch/x86/kvm/x86.c                 |  4 ++--
 6 files changed, 28 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -3
index 736129db272a..76ce691de2f8 100644
@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
-KVM_X86_OP_OPTIONAL(gmem_invalidate)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
 #endif
arch/x86/include/asm/kvm_host.h +3 -4
index e7df537d36d5..f60fec14c61a 100644
@@ -1724,11 +1724,10 @@ struct kvm_x86_ops {
 
        gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
        void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-       int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-                           kvm_pfn_t nr_pages, int max_order);
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+       int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                           kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
-       void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
        void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
arch/x86/kvm/svm/sev.c +18 -3
index adc58138fdd0..b00623b2f26a 100644
@@ -5114,8 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
        return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-                    kvm_pfn_t nr_pages, int max_order)
+static int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                                kvm_pfn_t nr_pages, int max_order)
 {
        struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
        kvm_pfn_t pfn_aligned;
@@ -5165,8 +5165,9 @@ int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
        return 0;
 }
 
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+static void sev_gmem_make_shared(kvm_pfn_t start, kvm_pfn_t nr_pages)
 {
+       kvm_pfn_t end = start + nr_pages;
        kvm_pfn_t pfn;
 
        if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
@@ -5228,6 +5229,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
        }
 }
 
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                    kvm_pfn_t nr_pages, int order, bool to_private)
+{
+       if (to_private) {
+               if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))
+                       return -EIO;
+
+               return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
+       }
+
+       sev_gmem_make_shared(pfn, nr_pages);
+       return 0;
+}
+
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        struct kvm_vcpu *vcpu;
arch/x86/kvm/svm/svm.c +1 -2
index 91286d46d13a..8c5018c65dc5 100644
@@ -5438,8 +5438,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
        .vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
        .vm_move_enc_context_from = sev_vm_move_enc_context_from,
 
-       .gmem_prepare = sev_gmem_prepare,
-       .gmem_invalidate = sev_gmem_invalidate,
+       .gmem_convert = sev_gmem_convert,
        .gmem_invalidate_range = sev_gmem_invalidate_range,
        .gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
arch/x86/kvm/svm/svm.h +2 -3
index e35419456648..57f692b5d709 100644
@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-                    kvm_pfn_t nr_pages, int max_order);
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                    kvm_pfn_t nr_pages, int max_order, bool to_private);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +2 -3
index d484d8b509b1..e509d50c247c 100644
@@ -10623,14 +10623,14 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
                          kvm_pfn_t nr_pages, int max_order)
 {
-       return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
+       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
 }
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-       kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+       WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
 }
 #endif
 
--

2.55.0.141.g00534a21ce-goog

#24 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

Smush x86's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin for SNP: they're invoked when private/shared memory is about to made accessible/visible to the guest or host. I.e. prepare() is really "make private", and invalidate() is really "make shared".

Quoted 8 lines

Using a single hook will yield more intuitive code when in-place conversion comes along.

For all intents and purposes, no functional change intended.

[...snip...]

I just tried this out with in-place conversion and I'm stuck trying to make a call to kvm_arch_gmem_convert() in the conversions ioctl.

I think we wanted to have a single .gmem_convert callback rather than .gmem_prepare and .gmem_reclaim, but it turns out that it's not quite easy to just call convert, because I need to pass both gfn and pfn to .gmem_convert().

The conversion is called on the inode:

__kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
size_t nr_pages, uint64_t attrs, pgoff_t *err_index)

Then guest_memfd would call

   kvm_gmem_convert(inode, start, end, to_private);

It is easy to get the kvm pointer for .gmem_convert(), but then to always pass something for gfn and pfn, I have to make sure that

+ the folio was allocated (to get a valid pfn)
+ some binding exists for the index (to get a valid gfn)

So for each folio that exists, I would check if there is a valid binding. If there is a valid binding, I compute the gfn, and if there's no valid binding I pass -1ull for gfn.

I think this is a little complicated and not actually all that intuitive...

Looking at KVM_AMD_SEV's Kconfig, it selects 4 gmem lifecycle callbacks:

config KVM_AMD_SEV
bool "AMD Secure Encrypted Virtualization (SEV) support" default y
depends on KVM_AMD && X86_64
depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM
select KVM_VM_MEMORY_ATTRIBUTES
select HAVE_KVM_ARCH_GMEM_CONVERT
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE
help
Provides support for launching encrypted VMs which use Secure Encrypted Virtualization (SEV), Secure Encrypted Virtualization with Encrypted State (SEV-ES), and Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP) technologies on AMD processors.

So actually smushing .gmem_prepare and .gmem_reclaim doesn't really save us CONFIG flags, and results in a kvm_gmem_convert() that needs to find the intersection of folio and binding existing just so the .gmem_convert call is valid for conversions in both directions.

If we set kvm_gmem_convert() up to do handle make_private() and make_shared() separately, then I don't think there's much value in collapsing .gmem_prepare() and .gmem_reclaim() just so we mux on the caller side and demux on the SNP side.

How about this:

1. Retain .gmem_reclaim() for "host reclaim" aka "return to host" aka "make shared" for x86.
+ Rename the make shared function to sev_gmem_reclaim
+ .gmem_reclaim = sev_gmem_reclaim
+ pKVM will defined kvm_arch_gmem_reclaim() to do what it needs
+ TDX and ARM CCA don't use this callback.
+ From __kvm_gmem_set_attributes(),

if (!to_private)
kvm_arch_gmem_reclaim(pfn, nr_pages)

+ From .free_folio() kvm_arch_gmem_reclaim(folio_pfn(folio), folio_nr_pages(folio))

2. Rename .gmem_prepare() to .gmem_grant(), or .gmem_assign(), something that is the opposite of "reclaim", that indicates the host granting/assigning/linking (other names welcome) the gfn <-> pfn
+ Only SNP will define this
+ .gmem_assign = sev_gmem_assign(kvm, gfn, pfn, nr_pages, order);
+ From kvm_gmem_get_pfn()

if (kvm_gmem_is_private_mem(inode, index))
kvm_arch_gmem_assign(kvm, gfn, pfn, nr_pages, order);

I think it is fair that "assign" or "grant" is only called for private pages, since only private pages need to be "given" away.

On the reclaim side, it's also fair that only shared pages need to be reclaimed.

And with this, we still have 4 CONFIGs just like above, except that it's now

select HAVE_KVM_ARCH_GMEM_ASSIGN
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE

And I guess it's more straightforward that one CONFIG selects one callback, avoiding having either of the CONFIGs select .gmem_convert():

Quoted 7 lines
arch/x86/include/asm/kvm_host.h +3 -0
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
[...snip...]
author maintainer NAK
#25 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

Quoted 31 lines · Ackerley Tng
Quoted 6 lines · Sean Christopherson

Smush x86's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin for SNP: they're invoked when private/shared memory is about to made accessible/visible to the guest or host. I.e. prepare() is really "make private", and invalidate() is really "make shared".

Quoted 8 lines

Using a single hook will yield more intuitive code when in-place conversion comes along.

For all intents and purposes, no functional change intended.

[...snip...]

I just tried this out with in-place conversion and I'm stuck trying to make a call to kvm_arch_gmem_convert() in the conversions ioctl.

I think we wanted to have a single .gmem_convert callback rather than .gmem_prepare and .gmem_reclaim, but it turns out that it's not quite easy to just call convert, because I need to pass both gfn and pfn to .gmem_convert().

The conversion is called on the inode:

__kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
size_t nr_pages, uint64_t attrs, pgoff_t *err_index)

Quoted 6 lines

Then guest_memfd would call

   kvm_gmem_convert(inode, start, end, to_private);

It is easy to get the kvm pointer for .gmem_convert(), but then to always pass something for gfn and pfn, I have to make sure that

As I stated somewhere else, conversion to SHARED does NOT and *should* NOT require a valid GFN. Conceptually, a SHARED page can't be strictly associated with a single GFN.

Quoted 29 lines

+ the folio was allocated (to get a valid pfn)
+ some binding exists for the index (to get a valid gfn)

So for each folio that exists, I would check if there is a valid binding. If there is a valid binding, I compute the gfn, and if there's no valid binding I pass -1ull for gfn.

I think this is a little complicated and not actually all that intuitive...

Looking at KVM_AMD_SEV's Kconfig, it selects 4 gmem lifecycle callbacks:

config KVM_AMD_SEV
bool "AMD Secure Encrypted Virtualization (SEV) support" default y
depends on KVM_AMD && X86_64
depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM
select KVM_VM_MEMORY_ATTRIBUTES
select HAVE_KVM_ARCH_GMEM_CONVERT
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE
help
Provides support for launching encrypted VMs which use Secure Encrypted Virtualization (SEV), Secure Encrypted Virtualization with Encrypted State (SEV-ES), and Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP) technologies on AMD processors.

Quoted 4 lines

So actually smushing .gmem_prepare and .gmem_reclaim doesn't really save us CONFIG flags, and results in a kvm_gmem_convert() that needs to find the intersection of folio and binding existing just so the .gmem_convert call is valid for conversions in both directions.

And I'm saying don't do that for the to_shared case. If passing a NULL @kvm and -1ull for @gfn from guest_memfd is too ugly, I'd be a-ok with providing separate kvm_arch_gmem_make_{private,shared}() under GMEM_CONVERT.

I guess at that point I don't have a strong preference betwee having a single kvm_x86_ops hook versus also having kvm_x86_ops.gmem_make_{private,shared}().

I.e. Option A:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		return kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false);
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
	}
	#endif

Or Option B:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
		return 0;
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
	}
	#endif

Typing it out, option B does look prettier...

Quoted 15 lines

If we set kvm_gmem_convert() up to do handle make_private() and make_shared() separately, then I don't think there's much value in collapsing .gmem_prepare() and .gmem_reclaim() just so we mux on the caller side and demux on the SNP side.

How about this:

1. Retain .gmem_reclaim() for "host reclaim" aka "return to host" aka "make shared" for x86.
+ Rename the make shared function to sev_gmem_reclaim
+ .gmem_reclaim = sev_gmem_reclaim
+ pKVM will defined kvm_arch_gmem_reclaim() to do what it needs
+ TDX and ARM CCA don't use this callback.
+ From __kvm_gmem_set_attributes(),

Quoted 2 lines

if (!to_private)
kvm_arch_gmem_reclaim(pfn, nr_pages)

NAK, pKVM doesn't want to "reclaim" on conversions to SHARED. reclaim() and convert() are two similar but distinct operations. Irrespective of pKVM, IMO it's also very useful to differentiate between "this page is being shared with the host, but KVM still owns the page" and "this page is being freed back to the host". E.g. reclaim() *must* succeed or take evasive action, whereas convert() can gracefully return an error.

Quoted 37 lines

+ From .free_folio() kvm_arch_gmem_reclaim(folio_pfn(folio), folio_nr_pages(folio))

2. Rename .gmem_prepare() to .gmem_grant(), or .gmem_assign(), something that is the opposite of "reclaim", that indicates the host granting/assigning/linking (other names welcome) the gfn <-> pfn
+ Only SNP will define this
+ .gmem_assign = sev_gmem_assign(kvm, gfn, pfn, nr_pages, order);
+ From kvm_gmem_get_pfn()

if (kvm_gmem_is_private_mem(inode, index))
kvm_arch_gmem_assign(kvm, gfn, pfn, nr_pages, order);

I think it is fair that "assign" or "grant" is only called for private pages, since only private pages need to be "given" away.

On the reclaim side, it's also fair that only shared pages need to be reclaimed.

And with this, we still have 4 CONFIGs just like above, except that it's now

select HAVE_KVM_ARCH_GMEM_ASSIGN
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE

And I guess it's more straightforward that one CONFIG selects one callback, avoiding having either of the CONFIGs select .gmem_convert():

Quoted 7 lines
arch/x86/include/asm/kvm_host.h +3 -0
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
[...snip...]
#26 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

Quoted 72 lines · Sean Christopherson
Quoted 31 lines · Ackerley Tng
Quoted 6 lines · Sean Christopherson

Smush x86's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin for SNP: they're invoked when private/shared memory is about to made accessible/visible to the guest or host. I.e. prepare() is really "make private", and invalidate() is really "make shared".

Quoted 8 lines

Using a single hook will yield more intuitive code when in-place conversion comes along.

For all intents and purposes, no functional change intended.

[...snip...]

I just tried this out with in-place conversion and I'm stuck trying to make a call to kvm_arch_gmem_convert() in the conversions ioctl.

I think we wanted to have a single .gmem_convert callback rather than .gmem_prepare and .gmem_reclaim, but it turns out that it's not quite easy to just call convert, because I need to pass both gfn and pfn to .gmem_convert().

The conversion is called on the inode:

__kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
size_t nr_pages, uint64_t attrs, pgoff_t *err_index)

Quoted 6 lines

Then guest_memfd would call

   kvm_gmem_convert(inode, start, end, to_private);

It is easy to get the kvm pointer for .gmem_convert(), but then to always pass something for gfn and pfn, I have to make sure that

As I stated somewhere else, conversion to SHARED does NOT and *should* NOT require a valid GFN. Conceptually, a SHARED page can't be strictly associated with a single GFN.

Quoted 29 lines

+ the folio was allocated (to get a valid pfn)
+ some binding exists for the index (to get a valid gfn)

So for each folio that exists, I would check if there is a valid binding. If there is a valid binding, I compute the gfn, and if there's no valid binding I pass -1ull for gfn.

I think this is a little complicated and not actually all that intuitive...

Looking at KVM_AMD_SEV's Kconfig, it selects 4 gmem lifecycle callbacks:

config KVM_AMD_SEV
bool "AMD Secure Encrypted Virtualization (SEV) support" default y
depends on KVM_AMD && X86_64
depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM
select KVM_VM_MEMORY_ATTRIBUTES
select HAVE_KVM_ARCH_GMEM_CONVERT
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE
help
Provides support for launching encrypted VMs which use Secure Encrypted Virtualization (SEV), Secure Encrypted Virtualization with Encrypted State (SEV-ES), and Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP) technologies on AMD processors.

Quoted 7 lines
Quoted 4 lines

So actually smushing .gmem_prepare and .gmem_reclaim doesn't really save us CONFIG flags, and results in a kvm_gmem_convert() that needs to find the intersection of folio and binding existing just so the .gmem_convert call is valid for conversions in both directions.

And I'm saying don't do that for the to_shared case. If passing a NULL @kvm and -1ull for @gfn from guest_memfd is too ugly,

It is ugly, but I thought the point of providing a single .gmem_convert callback was so the conversion path could just call convert with kvm and gfn arguments that work for both directions.

I like Option B below better, which should also resolve my issue.

Quoted 72 lines

I'd be a-ok with providing
separate kvm_arch_gmem_make_{private,shared}() under GMEM_CONVERT.

I guess at that point I don't have a strong preference betwee having a single kvm_x86_ops hook versus also having kvm_x86_ops.gmem_make_{private,shared}().

I.e. Option A:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		return kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false);
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
	}
	#endif

Or Option B:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
		return 0;
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
	}
	#endif

Typing it out, option B does look prettier...

Quoted 15 lines

If we set kvm_gmem_convert() up to do handle make_private() and make_shared() separately, then I don't think there's much value in collapsing .gmem_prepare() and .gmem_reclaim() just so we mux on the caller side and demux on the SNP side.

How about this:

1. Retain .gmem_reclaim() for "host reclaim" aka "return to host" aka "make shared" for x86.
+ Rename the make shared function to sev_gmem_reclaim
+ .gmem_reclaim = sev_gmem_reclaim
+ pKVM will defined kvm_arch_gmem_reclaim() to do what it needs
+ TDX and ARM CCA don't use this callback.
+ From __kvm_gmem_set_attributes(),

Quoted 2 lines

if (!to_private)
kvm_arch_gmem_reclaim(pfn, nr_pages)

Quoted 7 lines

NAK, pKVM doesn't want to "reclaim" on conversions to SHARED. reclaim() and convert() are two similar but distinct operations. Irrespective of pKVM, IMO it's also very useful to differentiate between "this page is being shared with the host, but KVM still owns the page" and "this page is being freed back to the host". E.g. reclaim() *must* succeed or take evasive action, whereas convert() can gracefully return an error.

Makes sense to distinguish reclaim and make_shared. Thanks!

Quoted 37 lines
Quoted 37 lines

+ From .free_folio() kvm_arch_gmem_reclaim(folio_pfn(folio), folio_nr_pages(folio))

2. Rename .gmem_prepare() to .gmem_grant(), or .gmem_assign(), something that is the opposite of "reclaim", that indicates the host granting/assigning/linking (other names welcome) the gfn <-> pfn
+ Only SNP will define this
+ .gmem_assign = sev_gmem_assign(kvm, gfn, pfn, nr_pages, order);
+ From kvm_gmem_get_pfn()

if (kvm_gmem_is_private_mem(inode, index))
kvm_arch_gmem_assign(kvm, gfn, pfn, nr_pages, order);

I think it is fair that "assign" or "grant" is only called for private pages, since only private pages need to be "given" away.

On the reclaim side, it's also fair that only shared pages need to be reclaimed.

And with this, we still have 4 CONFIGs just like above, except that it's now

select HAVE_KVM_ARCH_GMEM_ASSIGN
select HAVE_KVM_ARCH_GMEM_RECLAIM
select HAVE_KVM_ARCH_GMEM_INVALIDATE
select HAVE_KVM_ARCH_GMEM_POPULATE

And I guess it's more straightforward that one CONFIG selects one callback, avoiding having either of the CONFIGs select .gmem_convert():

Quoted 7 lines
arch/x86/include/asm/kvm_host.h +3 -0
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
[...snip...]
author maintainer
#27 lore raw

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

Quoted 2 lines · Ackerley Tng

Sean Christopherson <[email protected]> writes:
I like Option B below better, which should also resolve my issue.

Roger that. The more I stare at this, the more I like Option B, too. I'll go with that.

Quoted 51 lines
Quoted 51 lines

I'd be a-ok with providing
separate kvm_arch_gmem_make_{private,shared}() under GMEM_CONVERT.

I guess at that point I don't have a strong preference betwee having a single kvm_x86_ops hook versus also having kvm_x86_ops.gmem_make_{private,shared}().

I.e. Option A:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		return kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false);
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
	}
	#endif

Or Option B:

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
	int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
				kvm_pfn_t nr_pages, int max_order)
	{
		return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order);
	}
	int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
				bool to_private)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
		return 0;
	}
	#endif

	#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
	void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
	{
		kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
	}
	#endif
author maintainer
#28 lore raw

PATCH v5 7/7 KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

In reply to Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

No functional change intended.

---
 arch/x86/include/asm/kvm-x86-ops.h |  2 +-
 arch/x86/include/asm/kvm_host.h    |  2 +-
 arch/x86/kvm/Kconfig               |  2 +-
 arch/x86/kvm/x86.c                 |  8 ++++----
 include/linux/kvm_host.h           |  6 +++---
 virt/kvm/Kconfig                   |  2 +-
 virt/kvm/guest_memfd.c             | 12 ++++++------
 7 files changed, 17 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
index 76ce691de2f8..1764bad03b17 100644
@@ -146,7 +146,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/include/asm/kvm_host.h +1 -1
index f60fec14c61a..08c4af487fa1 100644
@@ -1724,7 +1724,7 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
arch/x86/kvm/Kconfig +1 -1
index e0e7ad015839..538ed1e80332 100644
@@ -160,7 +160,7 @@ config KVM_AMD_SEV
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
 	select KVM_GENERIC_MEMORY_ATTRIBUTES
-	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	select HAVE_KVM_ARCH_GMEM_POPULATE
arch/x86/kvm/x86.c +4 -4
index e509d50c247c..11017f49b94a 100644
@@ -10619,11 +10619,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return !kvm_arch_has_private_mem(kvm);
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private)
 {
-	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
+	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, to_private);
 }
 #endif
 
include/linux/kvm_host.h +3 -3
index 0a06234a8a8b..fdbd697d0337 100644
@@ -2579,9 +2579,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/Kconfig +1 -1
index 617876993225..c3c0ee253fc7 100644
@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD
        select XARRAY_MULTI
        bool
 
-config HAVE_KVM_ARCH_GMEM_PREPARE
+config HAVE_KVM_ARCH_GMEM_CONVERT
        bool
        depends on KVM_GUEST_MEMFD
 
virt/kvm/guest_memfd.c +6 -6
index 0f6745f3b919..173a7ed008d1 100644
@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  * On successful return the guest sees a zero page so as to avoid
  * leaking host data and the up-to-date flag is set.
  */
-static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				  gfn_t gfn, struct folio *folio)
+static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
+				 gfn_t gfn, struct folio *folio)
 {
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     nr_pages, folio_order(folio));
+	return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+				     nr_pages, folio_order(folio), true);
 #else
 	return 0;
 #endif
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		folio_mark_uptodate(folio);
 	}
 
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
 
 	folio_unlock(folio);
 
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Reviewed-by on v5 7/7
#29 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 11 lines · Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

No functional change intended.

Thanks!

Reviewed-by on v5 7/7
#30 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Quoted 6 lines

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

No functional change intended.

Cheers,
/fuad

Quoted 140 lines
---
 arch/x86/include/asm/kvm-x86-ops.h |  2 +-
 arch/x86/include/asm/kvm_host.h    |  2 +-
 arch/x86/kvm/Kconfig               |  2 +-
 arch/x86/kvm/x86.c                 |  8 ++++----
 include/linux/kvm_host.h           |  6 +++---
 virt/kvm/Kconfig                   |  2 +-
 virt/kvm/guest_memfd.c             | 12 ++++++------
 7 files changed, 17 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
index 76ce691de2f8..1764bad03b17 100644
@@ -146,7 +146,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/include/asm/kvm_host.h +1 -1
index f60fec14c61a..08c4af487fa1 100644
@@ -1724,7 +1724,7 @@ struct kvm_x86_ops {
 
        gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
        void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
        int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
                            kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
arch/x86/kvm/Kconfig +1 -1
index e0e7ad015839..538ed1e80332 100644
@@ -160,7 +160,7 @@ config KVM_AMD_SEV
        depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
        select ARCH_HAS_CC_PLATFORM
        select KVM_GENERIC_MEMORY_ATTRIBUTES
-       select HAVE_KVM_ARCH_GMEM_PREPARE
+       select HAVE_KVM_ARCH_GMEM_CONVERT
        select HAVE_KVM_ARCH_GMEM_RECLAIM
        select HAVE_KVM_ARCH_GMEM_INVALIDATE
        select HAVE_KVM_ARCH_GMEM_POPULATE
arch/x86/kvm/x86.c +4 -4
index e509d50c247c..11017f49b94a 100644
@@ -10619,11 +10619,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
        return !kvm_arch_has_private_mem(kvm);
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-                         kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                         kvm_pfn_t nr_pages, int max_order, bool to_private)
 {
-       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
+       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, to_private);
 }
 #endif
 
include/linux/kvm_host.h +3 -3
index 0a06234a8a8b..fdbd697d0337 100644
@@ -2579,9 +2579,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-                         kvm_pfn_t nr_pages, int max_order);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+                         kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/Kconfig +1 -1
index 617876993225..c3c0ee253fc7 100644
@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD
        select XARRAY_MULTI
        bool
 
-config HAVE_KVM_ARCH_GMEM_PREPARE
+config HAVE_KVM_ARCH_GMEM_CONVERT
        bool
        depends on KVM_GUEST_MEMFD
 
virt/kvm/guest_memfd.c +6 -7
index 0f6745f3b919..173a7ed008d1 100644
@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  * On successful return the guest sees a zero page so as to avoid
  * leaking host data and the up-to-date flag is set.
  */
-static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-                                 gfn_t gfn, struct folio *folio)
+static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
+                                gfn_t gfn, struct folio *folio)
 {
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
        unsigned long nr_pages = folio_nr_pages(folio);
        pgoff_t index;
 
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
        gfn = ALIGN_DOWN(gfn, nr_pages);
        index = kvm_gmem_get_index(slot, gfn);
 
-       return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-                                    nr_pages, folio_order(folio));
+       return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+                                    nr_pages, folio_order(folio), true);
 #else
        return 0;
 #endif
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
                folio_mark_uptodate(folio);
        }
 
-       r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+       r = kvm_gmem_make_private(kvm, slot, gfn, folio);
 
        folio_unlock(folio);
 
--

2.55.0.141.g00534a21ce-goog

#31 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

[...]

Quoted 12 lines
virt/kvm/guest_memfd.c +2 -2
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
  	gfn = ALIGN_DOWN(gfn, nr_pages);
  	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     nr_pages, folio_order(folio));
+	return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+				     nr_pages, folio_order(folio), true);
  #else
  	return 0;
  #endif

virt/kvm/guest_memfd.c int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 801

Quoted 5 lines
virt/kvm/guest_memfd.c +1 -1
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
  		folio_mark_uptodate(folio);
  	}
 
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);

I think this and above renaming don't make sense, because kvm_gmem_get_pfn() can be invoked for shared memory for non-Coco VMs when KVM_MEMSLOT_GMEM_ONLY is set.

Maybe this patch can be moved to gmem in-place series and after [1]?

[1] https://lore.kernel.org/all/[email protected]/

Quoted 2 lines
virt/kvm/guest_memfd.c +0 -0
  	folio_unlock(folio);
 
#32 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 27 lines · Xiaoyao Li
Quoted 8 lines · Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

[...]

Quoted 15 lines
(quoted diff) +2 -2
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
  	gfn = ALIGN_DOWN(gfn, nr_pages);
  	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     nr_pages, folio_order(folio));
+	return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+				     nr_pages, folio_order(folio), true);
  #else
  	return 0;
  #endif
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
  		folio_mark_uptodate(folio);
  	}
 
Quoted 7 lines

Review int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 801

Quoted 2 lines
(quoted diff) +1 -1
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);

I think this and above renaming don't make sense, because kvm_gmem_get_pfn() can be invoked for shared memory for non-Coco VMs when KVM_MEMSLOT_GMEM_ONLY is set.

Makes sense.

Quoted 5 lines

Maybe this patch can be moved to gmem in-place series and after [1]?

[1]
https://lore.kernel.org/all/[email protected]/

How about doing everything else in this patch in this patch, other than this:

virt/kvm/guest_memfd.c +0 -0
index 173a7ed008d17..9df6927dd8f55 100644
@@ -66,8 +66,8 @@ static pgoff_t kvm_gmem_get_index(struct

kvm_memory_slot *slot, gfn_t gfn)
* On successful return the guest sees a zero page so as to avoid
* leaking host data and the up-to-date flag is set.
*/
-static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn, struct folio *folio)
+static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn, struct folio *folio)
{
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
unsigned long nr_pages = folio_nr_pages(folio);
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
folio_mark_uptodate(folio);
}

- r = kvm_gmem_make_private(kvm, slot, gfn, folio);
+ r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

folio_unlock(folio);

I'll move just this renaming to [1] like you suggested.

I think it's okay to continue to always call prepare_folio(), and within the prepare_folio() function, only do conversion when the CONVERT CONFIG is defined.

I'll also move "Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role." from Sean's commit message to [1].

Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -0
  	folio_unlock(folio);
 
#33 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 75 lines · Ackerley Tng
Quoted 27 lines · Xiaoyao Li
Quoted 8 lines · Sean Christopherson

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in anticipation of supporting in-place conversion, at which point KVM will use the hook for both to-private and to-shared conversions, not just to "prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private() to better reflect its role.

[...]

Quoted 15 lines
(quoted diff) +2 -2
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
   	gfn = ALIGN_DOWN(gfn, nr_pages);
   	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     nr_pages, folio_order(folio));
+	return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+				     nr_pages, folio_order(folio), true);
   #else
   	return 0;
   #endif
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
   		folio_mark_uptodate(folio);
   	}
 
Quoted 7 lines

Review int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 801

Quoted 2 lines
(quoted diff) +1 -1
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);

I think this and above renaming don't make sense, because kvm_gmem_get_pfn() can be invoked for shared memory for non-Coco VMs when KVM_MEMSLOT_GMEM_ONLY is set.

Makes sense.

Quoted 5 lines

Maybe this patch can be moved to gmem in-place series and after [1]?

[1]
https://lore.kernel.org/all/[email protected]/

How about doing everything else in this patch in this patch, other than this:

virt/kvm/guest_memfd.c +2 -2
index 173a7ed008d17..9df6927dd8f55 100644
@@ -66,8 +66,8 @@ static pgoff_t kvm_gmem_get_index(struct
 kvm_memory_slot *slot, gfn_t gfn)
   * On successful return the guest sees a zero page so as to avoid
   * leaking host data and the up-to-date flag is set.
   */
-static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
-				 gfn_t gfn, struct folio *folio)
+static int kvm_gmem_prepare_folio(struct kvm *kvm, struct
 kvm_memory_slot *slot,
+				  gfn_t gfn, struct folio *folio)
  {
  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
  	unsigned long nr_pages = folio_nr_pages(folio);
virt/kvm/guest_memfd.c +1 -1
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct
 kvm_memory_slot *slot,
  		folio_mark_uptodate(folio);
  	}
 
-	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
+	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
 
  	folio_unlock(folio);
Quoted 5 lines

I'll move just this renaming to [1] like you suggested.

I think it's okay to continue to always call prepare_folio(), and within the prepare_folio() function, only do conversion when the CONVERT CONFIG is defined.

I don't think so.

This patch not only renames kvm_arch_gmem_prepare() to kvm_arch_gmem_convert(), but also adds one more parameter

'bool to_private'

and hardcodes the new parameter to true. This mean the arch callback will convert the folio to private unconditionally in kvm_prepare_folio() when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT is enabled.

Note the CONFIG is not a per-VM thing but a build time thing. The unconditionally-converting-to-private semantic can also be applied to gmem-only memslot for non-Coco VMs whenever the HAVE_KVM_ARCH_GMEM_CONVERT is enabled when building the kernel.

Though the code won't do anything for gmem-only memslot for non-Coco VMs, the literal semantic of the function and parameter is wrong.

author maintainer
#34 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 29 lines · Xiaoyao Li
Quoted 11 lines · Ackerley Tng

Xiaoyao Li <[email protected]> writes:

(quoted diff) +1 -1
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct
 kvm_memory_slot *slot,
  		folio_mark_uptodate(folio);
  	}
 
-	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
+	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
 
  	folio_unlock(folio);
Quoted 5 lines

I'll move just this renaming to [1] like you suggested.

I think it's okay to continue to always call prepare_folio(), and within the prepare_folio() function, only do conversion when the CONVERT CONFIG is defined.

I don't think so.

This patch not only renames kvm_arch_gmem_prepare() to kvm_arch_gmem_convert(), but also adds one more parameter

'bool to_private'

and hardcodes the new parameter to true. This mean the arch callback will convert the folio to private unconditionally in kvm_prepare_folio() when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT is enabled.

Quoted 7 lines

Note the CONFIG is not a per-VM thing but a build time thing. The unconditionally-converting-to-private semantic can also be applied to gmem-only memslot for non-Coco VMs whenever the HAVE_KVM_ARCH_GMEM_CONVERT is enabled when building the kernel.

Though the code won't do anything for gmem-only memslot for non-Coco VMs, the literal semantic of the function and parameter is wrong.

Agreed. How about I add a patch to guard the call with:

	if (kvm_arch_has_private_mem(kvm) &&
	    !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

which is semantically correct pre-in-place conversion. And then when in-place conversion comes along, that will get switched to:

	if (kvm_gmem_is_private_mem(inode, index))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

Which IMO yields a very clean and intuitive diff.

Actually, even better would be to insert a patch to provide kvm_gmem_is_private_mem() and kvm_gmem_is_shared_mem() as part of this prep, and pull in "KVM: guest_memfd:
Only prepare folios for private pages" with a massaged shortlog+changelog.

I.e. have this at the end of this prep work:

static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index)

{
	return kvm_arch_has_private_mem(kvm) && 
	       !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED);
}

static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)

{
	return !kvm_gmem_is_private_mem(inode, index);
}
	if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff))
		return VM_FAULT_SIGBUS;

	if (kvm_gmem_is_private_mem(inode, index))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

And then "KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings" isn't changing the semantics of the callers, it's only changing the internal plumbing for PRIVATE vs. SHARED.

#35 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 47 lines · Sean Christopherson
Quoted 29 lines · Xiaoyao Li
Quoted 11 lines · Ackerley Tng

Xiaoyao Li <[email protected]> writes:

(quoted diff) +1 -1
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct
 kvm_memory_slot *slot,
   		folio_mark_uptodate(folio);
   	}
 
-	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
+	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
 
   	folio_unlock(folio);
Quoted 5 lines

I'll move just this renaming to [1] like you suggested.

I think it's okay to continue to always call prepare_folio(), and within the prepare_folio() function, only do conversion when the CONVERT CONFIG is defined.

I don't think so.

This patch not only renames kvm_arch_gmem_prepare() to kvm_arch_gmem_convert(), but also adds one more parameter

'bool to_private'

and hardcodes the new parameter to true. This mean the arch callback will convert the folio to private unconditionally in kvm_prepare_folio() when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT is enabled.

Quoted 7 lines

Note the CONFIG is not a per-VM thing but a build time thing. The unconditionally-converting-to-private semantic can also be applied to gmem-only memslot for non-Coco VMs whenever the HAVE_KVM_ARCH_GMEM_CONVERT is enabled when building the kernel.

Though the code won't do anything for gmem-only memslot for non-Coco VMs, the literal semantic of the function and parameter is wrong.

Agreed. How about I add a patch to guard the call with:

	if (kvm_arch_has_private_mem(kvm) &&
	    !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

which is semantically correct pre-in-place conversion. And then when in-place conversion comes along, that will get switched to:

Quoted 8 lines
	if (kvm_gmem_is_private_mem(inode, index))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

Which IMO yields a very clean and intuitive diff.

Actually, even better would be to insert a patch to provide kvm_gmem_is_private_mem() and kvm_gmem_is_shared_mem() as part of this prep, and pull in "KVM: guest_memfd:
Only prepare folios for private pages" with a massaged shortlog+changelog.

yeah. This looks good!

Quoted 7 lines

I.e. have this at the end of this prep work:

static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index)

{
	return kvm_arch_has_private_mem(kvm) &&
	       !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED);
}

I'm wondering if we really need to check kvm_arch_has_private_mem() here. It looks checking GUEST_MEMFD_FLAG_INIT_SHARED is sufficient.

Quoted 15 lines

static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index)

{
	return !kvm_gmem_is_private_mem(inode, index);
}
	if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff))
		return VM_FAULT_SIGBUS;

	if (kvm_gmem_is_private_mem(inode, index))
		r = kvm_gmem_make_private(kvm, slot, gfn, folio);

And then "KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings" isn't changing the semantics of the callers, it's only changing the internal plumbing for PRIVATE vs. SHARED.

author maintainer
#36 lore raw

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

Quoted 11 lines · Xiaoyao Li
Quoted 7 lines · Sean Christopherson

I.e. have this at the end of this prep work:

static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index)

{
	return kvm_arch_has_private_mem(kvm) &&
	       !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED);
}

I'm wondering if we really need to check kvm_arch_has_private_mem() here. It looks checking GUEST_MEMFD_FLAG_INIT_SHARED is sufficient.

Ya, I came to the same conclusion and ended up dropping that line so that the patch would be a guaranteed "No functional change intended".