KVM: SEV: Fix RMP #PF due to freeing in-use VMSA
Subsystems (4)
Sean Christopherson
maintainerv4 cover KVM: SEV: Fix RMP #PF due to freeing in-use VMSA
Rework KVM's handling of guest-provided (and always guest_memfd-backed) VMSAs to forcefully reclaim VMSA pages when the pages are being freed from their backing gmem instance, e.g. in response to PUNCH_HOLE. In the worst case scenario, marking the page SHARED in the RMP will fail due to the page being IN_USE, ultimately leading to RMP #PF violations due to guest_memfd freeing the memory back to the kernel while it's still assigned to a VM.
Note, the implementation nearly identical to that used by KVM for VMX's APIC access page (which isn't guest controlled, but is migratable and whose PA is shoved directly into a vCPU control structure).
On the backend of the series, rework prepare()+invalidate() into a single convert() hook, to eliminate the misleading use of the term "invalidate", to (obviously) drop yet another callback, and to prepare for in-place conversion, e.g. so that KVM ends up with something like:
if (!to_private) kvm_gmem_make_shared(inode, start, end);
instead of a funky:
if (!to_private) kvm_gmem_invalidate(inode, start, end);
v4:
- Collect reviews. [Mike, Ackerley]
- Rework prepare()+invalidate() into a common convert() instead of simply renaming invalidate() to reclaim_memory(). [Ackerley]
- Fix several (mostly benign) issues in the "prepare" flow.
v3:
- https://lore.kernel.org/all/[email protected]
- Ensure disabling quirks while the VM is live won't result in KVM skipping the back-half of "zap all fast". [Sashiko]
- s/gmem_free_folio/gmem_reclaim_memory. [Ackerley]
- Use {READ,WRITE}_ONCE() when accessing the guest's VMSA GPA outside of the per-vCPU mutex, and comment. [Sashiko]
v2:
- https://lore.kernel.org/all/[email protected]
- Invalidate VMSAs if the memslot is DELETED or MOVED. [Sashiko]
- Limit stable@ patches without a Fixes to 6.12+.
v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (18):
KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP
guests
KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if
CONFIG_KVM_AMD_SEV=y
KVM: x86: Serialize writes to disabled_quirks using kvm->lock
KVM: x86: Ensure runtime reads of disabled_quirks are resolved once
KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into
kvm_arch_flush_shadow_memslot()
KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back"
halves
KVM: x86/mmu: Use split "zap all fast" helpers when invalidating
memslot
KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is
zapped
KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid
KVM: x86: Guard .gmem_prepare() declarations with
HAVE_KVM_GMEM_PREPARE=y
KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare()
failure
KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
KVM: guest_memfd: Explicitly pass number of pages to
kvm_arch_gmem_prepare()
KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a
folio
KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into
.gmem_convert()
arch/x86/include/asm/kvm-x86-ops.h | 9 +- arch/x86/include/asm/kvm_host.h | 12 +- arch/x86/kvm/Kconfig | 2 +- arch/x86/kvm/mmu/mmu.c | 74 +++++++----- arch/x86/kvm/svm/sev.c | 186 ++++++++++++++++++++++------- arch/x86/kvm/svm/svm.c | 9 +- arch/x86/kvm/svm/svm.h | 19 +-- arch/x86/kvm/x86.c | 21 ++-- arch/x86/kvm/x86.h | 2 +- include/linux/kvm_host.h | 7 +- virt/kvm/Kconfig | 2 +- virt/kvm/guest_memfd.c | 53 ++++---- 12 files changed, 256 insertions(+), 140 deletions(-)
base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 1/18 KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
Track the GPA of the guest-provided VMSA used after AP_CREATION events when running SNP guests, instead of simply tracking whether or not the vCPU is using a guest-provided VMSA. KVM needs to know the GPA of the VMSA that's actively being used so that it can react to MMU invalidation events, i.e. so that KVM can drop the VMSA if its backing guest_memfd page is punched out of existence.
Opportunistically rename snp_vmsa_gpa to clarify that it tracks the pending VMSA GPA, whereas snp_guest_vmsa_gpa now tracks the in-use VMSA GPA.
Note! Take care to track the GPA, not the GFN, as VALID_PAGE() won't behave correctly if an invalid GFN is converted to a GPA for checking.
Note #2! Keep snp_has_guest_vmsa so that switching to a guest-provided VMSA is sticky, even if the guest-provided VMSA becomes invalid.
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 14 +++++++++----- arch/x86/kvm/svm/svm.h | 3 ++- 2 files changed, 11 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/sev.c +9 -5
@@ -4003,6 +4003,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)/* Clear use of the VMSA */ svm->vmcb->control.vmsa_pa =INVALID_PAGE ; + svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ;/* * When replacing the VMSA during SEV-SNP AP creation,
@@ -4010,11 +4011,11 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (svm->sev_es.snp_vmsa_gpa)) +if (!VALID_PAGE (svm->sev_es.snp_pending_vmsa_gpa))return ; - gfn =gpa_to_gfn (svm->sev_es.snp_vmsa_gpa); - svm->sev_es.snp_vmsa_gpa =INVALID_PAGE ; + gfn =gpa_to_gfn (svm->sev_es.snp_pending_vmsa_gpa); + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; slot =gfn_to_memslot (vcpu->kvm, gfn);if (!slot)
@@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ + svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);/* Mark the vCPU as runnable */
@@ -4105,10 +4107,10 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm)return -EINVAL ; } - target_svm->sev_es.snp_vmsa_gpa = svm->vmcb->control.exit_info_2; + target_svm->sev_es.snp_pending_vmsa_gpa = svm->vmcb->control.exit_info_2;break ;case SVM_VMGEXIT_AP_DESTROY : - target_svm->sev_es.snp_vmsa_gpa =INVALID_PAGE ; + target_svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ;break ;default : vcpu_unimpl(vcpu, "vmgexit: invalid AP creation request [%#x] from guest\n",
@@ -4791,6 +4793,8 @@ int sev_vcpu_create(struct kvm_vcpu *vcpu)return -ENOMEM ; svm->sev_es.vmsa =page_address (vmsa_page); + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; + svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ; vcpu->arch.guest_tsc_protected =snp_is_secure_tsc_enabled (vcpu->kvm);
arch/x86/kvm/svm/svm.h +2 -1
@@ -271,7 +271,8 @@ struct vcpu_sev_es_state {u64 ghcb_registered_gpa;struct mutex snp_vmsa_mutex;/* Used to handle concurrent updates of VMSA. */ -gpa_t snp_vmsa_gpa; +gpa_t snp_pending_vmsa_gpa; +gpa_t snp_guest_vmsa_gpa;bool snp_ap_waiting_for_reset;bool snp_has_guest_vmsa; };
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] Missing MMU invalidation retry protocol when resolving and installing the guest_memfd VMSA page, causing a TOCTOU race leading to physical Use-After-Free. --
--- Patch [1]: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests --- commit 78360a5e270c82d6286a0a511ee0ec1f25aec337
Author: Sean Christopherson <[email protected]>
KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
This commit tracks the GPA of the guest-provided VMSA used after AP_CREATION events for SNP guests. It renames the tracking variables to clarify pending vs. in-use GPAs to facilitate reacting to MMU invalidation events.
Quoted 4 lines
arch/x86/kvm/svm/sev.c +0
-0
[ ... ]
arch/x86/kvm/svm/sev.c static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) · line 4044
Quoted 6 lines
arch/x86/kvm/svm/sev.c +1
-0@@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ + svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);
[Severity: Critical]
This is a pre-existing issue, but does this bypass the MMU invalidation retry protocol?
Looking at sev_snp_init_protected_guest_state(), KVM resolves the PFN via kvm_gmem_get_pfn(), directly installs it into the hardware VMCB, and drops the page reference:
if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL)) return; ... svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn); ... kvm_release_page_clean(page);
Can this race with a concurrent guest_memfd hole punch if kvm->mmu_lock isn't held and the invalidation sequence counter isn't verified?
If the page is freed back to the host allocator, could the hardware continue to use a freed host physical page for the VMSA, leading to a physical use-after-free?
Quoted 2 lines
arch/x86/kvm/svm/sev.c +0
-0/* Mark the vCPU as runnable */ kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE );
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerRe: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
Quoted 27 lines · On Thu, Jul 09, 2026, [email protected] wrote:
Quoted 4 lines
arch/x86/kvm/svm/sev.c +0
-0[ ... ]
arch/x86/kvm/svm/sev.c static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) · line 4044
Quoted 6 lines
arch/x86/kvm/svm/sev.c +1
-0@@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ + svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);[Severity: Critical]
This is a pre-existing issue, but does this bypass the MMU invalidation retry protocol?Looking at
sev_snp_init_protected_guest_state(), KVM resolves the PFN viakvm_gmem_get_pfn(), directly installs it into the hardware VMCB, and drops the page reference:if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL)) return; ... svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn); ... kvm_release_page_clean(page);
Quoted 6 lines
Can this race with a concurrent
guest_memfdhole punch ifkvm->mmu_lockisn't held and the invalidation sequence counter isn't verified?If the page is freed back to the host allocator, could the hardware continue to use a freed host physical page for the VMSA, leading to a physical use-after-free?
Read the cover letter?
Sean Christopherson
maintainerPATCH v4 2/18 KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
In reply to Sean ChristophersonExtract the loading/retrieval of a guest-provided VMSA to a separate helper so that KVM can reuse the core logic when refreshing the VMSA after an MMU invalidation from guest_memfd.
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 52 +++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 21 deletions(-)
arch/x86/kvm/svm/sev.c +31 -21
@@ -3979,29 +3979,17 @@ static int snp_begin_psc(struct vcpu_svm *svm)return snp_do_psc (svm); } -/* -* Invoked as part of svm_vcpu_reset() processing of an init event. -*/ -static void sev_snp_init_protected_guest_state (struct kvm_vcpu *vcpu) +static void sev_snp_reload_vmsa (struct kvm_vcpu *vcpu,gpa_t gpa) {struct vcpu_svm *svm =to_svm (vcpu);struct kvm_memory_slot *slot; +gfn_t gfn =gpa_to_gfn (gpa);struct page *page;kvm_pfn_t pfn; -gfn_t gfn; -guard (mutex)(&svm->sev_es.snp_vmsa_mutex); +lockdep_assert_held (&svm->sev_es.snp_vmsa_mutex); -if (!svm->sev_es.snp_ap_waiting_for_reset) -return ; - - svm->sev_es.snp_ap_waiting_for_reset = false; - -/* Mark the vCPU as offline and not runnable */ - vcpu->arch.pv.pv_unhalted = false; -kvm_set_mp_state (vcpu,KVM_MP_STATE_HALTED ); - -/* Clear use of the VMSA */ +/* Clear use of the VMSA. */ svm->vmcb->control.vmsa_pa =INVALID_PAGE ; svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ;
@@ -4011,12 +3999,9 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (svm->sev_es.snp_pending_vmsa_gpa)) +if (!VALID_PAGE (gpa))return ; - gfn =gpa_to_gfn (svm->sev_es.snp_pending_vmsa_gpa); - svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; - slot =gfn_to_memslot (vcpu->kvm, gfn);if (!slot)return ;
@@ -4040,7 +4025,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ - svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); + svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);/* Mark the vCPU as runnable */
@@ -4054,6 +4039,31 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)kvm_release_page_clean (page); } +/* +* Invoked as part of svm_vcpu_reset() processing of an init event. +*/ +static void sev_snp_init_protected_guest_state (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); +gpa_t gpa; + +guard (mutex)(&svm->sev_es.snp_vmsa_mutex); + +if (!svm->sev_es.snp_ap_waiting_for_reset) +return ; + + svm->sev_es.snp_ap_waiting_for_reset = false; + +/* Mark the vCPU as offline and not runnable */ + vcpu->arch.pv.pv_unhalted = false; +kvm_set_mp_state (vcpu,KVM_MP_STATE_HALTED ); + + gpa = svm->sev_es.snp_pending_vmsa_gpa; + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; + +sev_snp_reload_vmsa (vcpu, gpa); +} +static int sev_snp_ap_creation (struct vcpu_svm *svm) {struct kvm_sev_info *sev =to_kvm_sev_info (svm->vcpu.kvm);
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [Critical] Missing MMU invalidation sequence check and invalidation hooks when resolving and installing the guest memory PFN into the hardware VMCB (vmsa_pa), allowing a guest to trigger a host Use-After-Free via hole-punching.
- [High] Early return on failure to retrieve or load the VMSA PFN leaves the vCPU permanently in the KVM_MP_STATE_HALTED state, causing KVM_RUN to hang indefinitely instead of returning an error to userspace. --
--- Patch [2]: [PATCH v4 02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper --- commit 77b2286f40978a436ebf37446c67270e552074af
Author: Sean Christopherson <[email protected]>
KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
Extract the loading/retrieval of a guest-provided VMSA to a separate helper so that KVM can reuse the core logic when refreshing the VMSA after an MMU invalidation from guest_memfd.
Quoted 4 lines
arch/x86/kvm/svm/sev.c +0
-0
[ ... ]
Quoted 4 lines
arch/x86/kvm/svm/sev.c +0
-0@@ -4011,12 +3999,9 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)*/ vmcb_mark_all_dirty (svm->vmcb);
arch/x86/kvm/svm/sev.c static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) · line 4007
Quoted 10 lines
arch/x86/kvm/svm/sev.c +1
-4-if (!VALID_PAGE (svm->sev_es.snp_pending_vmsa_gpa)) +if (!VALID_PAGE (gpa))return ; - gfn =gpa_to_gfn (svm->sev_es.snp_pending_vmsa_gpa); - svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; - slot =gfn_to_memslot (vcpu->kvm, gfn);if (!slot)return ;
[Severity: High]
This is a pre-existing issue, but if the VMSA loading fails here (e.g., due to an invalid GPA or missing memslot), does the vCPU remain permanently locked in KVM_MP_STATE_HALTED?
Since sev_snp_init_protected_guest_state() marks the vCPU as HALTED before calling this reload helper, returning early bypasses the eventual transition to KVM_MP_STATE_RUNNABLE. Could this cause KVM_RUN to hang indefinitely instead of safely returning an error to userspace?
[ ... ]
arch/x86/kvm/svm/sev.c static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) · line 4029
Quoted 7 lines
arch/x86/kvm/svm/sev.c +1
-1@@ -4040,7 +4025,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ - svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); + svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);
[Severity: Critical]
This isn't a bug introduced by this patch, but does resolving and installing the PFN directly into the hardware VMCB (vmsa_pa) without using the mmu_invalidate_seq retry protocol allow a guest to trigger a host use-after-free?
If an untrusted guest triggers AP creation with a guest_memfd-backed VMSA, and then punches a hole in that guest_memfd, could KVM miss the concurrent invalidation since this path doesn't check the invalidation sequence under kvm->mmu_lock? The hardware might continue using and writing to the freed page.
Quoted 5 lines
arch/x86/kvm/svm/sev.c +0
-0/* Mark the vCPU as runnable */ @@ -4054,6 +4039,31 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)kvm_release_page_clean (page); }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Sean Christopherson
maintainerPATCH v4 3/18 KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
In reply to Sean ChristophersonAlways mark the vCPU as RUNNABLE after responding to AP_CREATE, even if the guest-specified VMSA is unusable, e.g. isn't backed by a memslot or doesn't have a backing guest_memfd page. If the VMSA is unusable, leaving the vCPU in a non-running state will effectively hang the vCPU instead of reporting an error to userspace. This will also allow retrying the VMSA load in the future, to fix a bug where KVM doesn't honor guest_memfd invalidation events, e.g. if AP_CREATION races with PUNCH_HOLE.
--- arch/x86/kvm/svm/sev.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/sev.c +9 -3
@@ -4028,9 +4028,6 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn); -/* Mark the vCPU as runnable */ -kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE ); -/* * gmem pages aren't currently migratable, but if this ever changes * then care should be taken to ensure svm->sev_es.vmsa is pinned
@@ -4062,6 +4059,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ;sev_snp_reload_vmsa (vcpu, gpa); + +/* +* Mark the vCPU as runnable for CREATE requests, indicated by a valid +* VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will +* fail instead of blocking indefinitely and hanging the vCPU, e.g. if +* the backing guest_memfd page is unavailable. +*/ +if (VALID_PAGE (gpa)) +kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE ); }static int sev_snp_ap_creation (struct vcpu_svm *svm)
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] KVM incorrectly executes VMRUN with a dummy unmeasured host VMSA instead of failing KVM_RUN when a vCPU's initial AP_CREATE VMSA load fails.
Pre-existing issues:
- [Critical] sev_snp_reload_vmsa() bypasses the MMU notifier invalidation retry protocol, leading to a Use-After-Free of a physical page if a concurrent guest_memfd invalidation occurs. --
--- Patch [3]: [PATCH v4 03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable --- commit ca23b8314664c0ed691814678164a510162e2299
Author: Sean Christopherson <[email protected]>
KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
This commit marks a vCPU as RUNNABLE after responding to an AP_CREATE request, even if the guest-specified VMSA is unusable. This prevents the vCPU from hanging in a non-running state and instead allows KVM_RUN to return an error to userspace. It also enables retrying the VMSA load in the future.
arch/x86/kvm/svm/sev.c static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) · line 4029
Quoted 7 lines
arch/x86/kvm/svm/sev.c +0
-0@@ -4028,9 +4028,6 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);
[Severity: Critical]
This is a pre-existing issue, but does sev_snp_reload_vmsa() bypass the MMU notifier invalidation retry protocol here?
If a concurrent PUNCH_HOLE occurs on the guest_memfd while sev_snp_reload_vmsa() fetches and installs a guest VMSA, this could lead to a use-after-free of the physical page.
sev_snp_reload_vmsa() resolves the PFN via kvm_gmem_get_pfn() and directly installs it into svm->vmcb->control.vmsa_pa without holding kvm->mmu_lock or checking mmu_invalidate_retry_gfn(). It then drops the page reference.
If the page is subsequently freed by guest_memfd, KVM would retain the stale physical address in vmsa_pa, and the hardware could write guest state to the freed page during VMRUN.
Quoted 3 lines
arch/x86/kvm/svm/sev.c +0
-3-/* Mark the vCPU as runnable */ -kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE ); -
[ ... ]
arch/x86/kvm/svm/sev.c static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) · line 4070
Quoted 13 lines
arch/x86/kvm/svm/sev.c +9
-0@@ -4062,6 +4059,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ;sev_snp_reload_vmsa (vcpu, gpa); + +/* +* Mark the vCPU as runnable for CREATE requests, indicated by a valid +* VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will +* fail instead of blocking indefinitely and hanging the vCPU, e.g. if +* the backing guest_memfd page is unavailable. +*/ +if (VALID_PAGE (gpa)) +kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE );
[Severity: High]
Does this allow KVM to incorrectly execute VMRUN with a dummy unmeasured host VMSA when the initial AP_CREATE VMSA load fails?
If a guest issues an AP_CREATE request with an invalid VMSA (such as having no backing guest_memfd page) for a vCPU that has not yet successfully loaded a guest VMSA, marking it RUNNABLE is intended to make pre_sev_run() fail KVM_RUN.
However, because snp_has_guest_vmsa remains false on failure, sev_es_init_vmcb() overwrites the INVALID_PAGE state with the KVM-allocated fallback VMSA physical address:
if (!svm->sev_es.snp_has_guest_vmsa) {
if (svm->sev_es.vmsa)
svm->vmcb->control.vmsa_pa = __pa(svm->sev_es.vmsa);
else
svm->vmcb->control.vmsa_pa = INVALID_PAGE;
}Because of this, pre_sev_run() will see a valid vmsa_pa and pass:
if (is_sev_es_guest(vcpu) && !VALID_PAGE(svm->vmcb->control.vmsa_pa)) return -EINVAL;
Will this result in a hardware RMP violation or guest state corruption when KVM attempts VMRUN with the dummy VMSA?
Quoted 1 line
arch/x86/kvm/svm/sev.c +0
-0}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Sean Christopherson
maintainerPATCH v4 4/18 KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y
In reply to Sean ChristophersonWire up the SEV-SNP guest_memfd kvm_x86_ops hooks if and only if SEV is actually enabled, and drop the now-unnecessary stubs. Leaving the hooks NULL allows the static call infrastructure to elide the CALL+RET, and more importantly, referencing the hooks if and only if SEV support is enabled will allow conditionally definining the hooks using their corresponding HAVE_KVM_ARCH_GMEM_XXX Kconfig.
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 8 ++++---- arch/x86/kvm/svm/svm.h | 10 ---------- 2 files changed, 4 insertions(+), 14 deletions(-)
arch/x86/kvm/svm/svm.c +4 -4
@@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, + + .gmem_prepare = sev_gmem_prepare, + .gmem_invalidate = sev_gmem_invalidate, + .gmem_max_mapping_level = sev_gmem_max_mapping_level, #endif .check_emulate_instruction = svm_check_emulate_instruction,
@@ -5459,10 +5463,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, .alloc_apic_backing_page = svm_alloc_apic_backing_page, - - .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, - .gmem_max_mapping_level = sev_gmem_max_mapping_level, };/*
arch/x86/kvm/svm/svm.h +0 -10
@@ -1035,16 +1035,6 @@ static inline int sev_cpu_init(struct svm_cpu_data *sd) { return 0; }static inline int sev_dev_get_attr (u32 group,u64 attr,u64 *val) {return -ENXIO ; }#define max_sev_asid 0static inline void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code) {} -static inline int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order) -{ -return 0 ; -} -static inline void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) {} -static inline int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private) -{ -return 0 ; -} -static inline struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu) {return NULL ;
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y
Quoted 21 lines · Sean Christopherson
Wire up the SEV-SNP
guest_memfdkvm_x86_opshooks if and only if SEV is actually enabled, and drop the now-unnecessary stubs. Leaving the hooks NULL allows the static call infrastructure to elide the CALL+RET, and more importantly, referencing the hooks if and only if SEV support is enabled will allow conditionally definining the hooks using their correspondingHAVE_KVM_ARCH_GMEM_XXXKconfig.No functional change intended.
Cc [email protected] # 6.12.xSigned-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/svm.c | 8 ++++---- arch/x86/kvm/svm/svm.h | 10 ---------- 2 files changed, 4 insertions(+), 14 deletions(-)arch/x86/kvm/svm/svm.c +0
-0
arch/x86/kvm/svm/svm.c struct kvm_x86_ops svm_x86_ops __initdata = { · line 5452
Quoted 5 lines
arch/x86/kvm/svm/svm.c +2
-0@@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, + + .gmem_prepare = sev_gmem_prepare,
Here .gmem_prepare assignment is gated on CONFIG_KVM_AMD_SEV but later the definition of .gmem_prepare is gated on CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE.
CONFIG_KVM_AMD_SEV selects CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE, but I wonder if there's a chance due to config disconnects this will cause compilation issues? Like .gmem_prepare will get assigned but it was not declared? Or is that actually intentional?
Quoted 19 lines
arch/x86/kvm/svm/svm.c +2
-4+ .gmem_invalidate = sev_gmem_invalidate, + .gmem_max_mapping_level = sev_gmem_max_mapping_level, #endif .check_emulate_instruction = svm_check_emulate_instruction,@@ -5459,10 +5463,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, .alloc_apic_backing_page = svm_alloc_apic_backing_page, - - .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, - .gmem_max_mapping_level = sev_gmem_max_mapping_level, };/*
Sean Christopherson
maintainerRe: [PATCH v4 04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y
Quoted 33 lines · Ackerley Tng
Quoted 21 lines · Sean Christopherson
Wire up the SEV-SNP
guest_memfdkvm_x86_opshooks if and only if SEV is actually enabled, and drop the now-unnecessary stubs. Leaving the hooks NULL allows the static call infrastructure to elide the CALL+RET, and more importantly, referencing the hooks if and only if SEV support is enabled will allow conditionally definining the hooks using their correspondingHAVE_KVM_ARCH_GMEM_XXXKconfig.No functional change intended.
Cc [email protected] # 6.12.xSigned-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/svm.c | 8 ++++---- arch/x86/kvm/svm/svm.h | 10 ---------- 2 files changed, 4 insertions(+), 14 deletions(-)arch/x86/kvm/svm/svm.c +0
-0arch/x86/kvm/svm/svm.c struct kvm_x86_ops svm_x86_ops __initdata = { · line 5452
Quoted 5 lines
arch/x86/kvm/svm/svm.c +2
-0@@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, + + .gmem_prepare = sev_gmem_prepare,Here .gmem_prepare assignment is gated on
CONFIG_KVM_AMD_SEVbut later the definition of .gmem_prepare is gated onCONFIG_HAVE_KVM_ARCH_GMEM_PREPARE.
Quoted 4 lines
CONFIG_KVM_AMD_SEVselectsCONFIG_HAVE_KVM_ARCH_GMEM_PREPARE, but I wonder if there's a chance due to config disconnects this will cause compilation issues? Like .gmem_prepare will get assigned but it was not declared? Or is that actually intentional?
Intentional. If we end up with KVM_AMD_SEV=y, but HAVE_KVM_ARCH_GMEM_PREPARE=n, then I definitely want the build to fail.
Sean Christopherson
maintainerPATCH v4 5/18 KVM: x86: Serialize writes to disabled_quirks using kvm->lock
In reply to Sean ChristophersonProtect writes to disabled_quirks with kvm->lock to ensure KVM doesn't clobber state in the unlikely scenario that userspace disables disparate quirks from multiple tasks. More importantly, this will allow wrapping accesses with {READ,WRITE}_ONCE without "needing" to also guard the writer with a useless and confusing READ_ONCE (since the RMW wouldn't be atomic anyways).
Ideally, KVM would disallow disabling quirks once quirks are "live", but that would be a potentially breaking userspace ABI change, and while all existing quirks are fully live only after vCPUs have been created, several MMU-related quirks, IGNORE_GUEST_PAT and SLOT_ZAP_ALL, are partially live at all times. Because populating MMUs requires a vCPU, the guest-visible behavior of IGNORE_GUEST_PAT and SLOT_ZAP_ALL requires a vCPU, but for KVM itself, processing the quirk (or not) has functional impact, i.e. for all intents and purposes, KVM can't prevent those quirks from being disabled after they've been consumed.
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -3939,7 +3939,9 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,break ; fallthrough;case KVM_CAP_DISABLE_QUIRKS : +mutex_lock (&kvm->lock); kvm->arch.disabled_quirks |= cap->args[0 ] & kvm_caps.supported_quirks; +mutex_unlock (&kvm->lock); r =0 ;break ;case KVM_CAP_SPLIT_IRQCHIP : {
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 6/18 KVM: x86: Ensure runtime reads of disabled_quirks are resolved once
In reply to Sean ChristophersonWrap the sole reader of disabled_quirks with READ_ONCE(), and wrap the post-VM-creation write to disabled_quirks with WRITE_ONCE(), to ensure checking the status of a quirk doesn't re-read disabled_quirks *if* the caller needs such a guarantee. This will allow splitting the "fast" MMU zap into front and back halves, without potentially skipping the back half if SLOT_ZAP_ALL were concurrently disabled (which would be "fine" in the current code base, but far from ideal).
--- arch/x86/kvm/x86.c | 3 ++- arch/x86/kvm/x86.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +2 -1
@@ -3940,7 +3940,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, fallthrough;case KVM_CAP_DISABLE_QUIRKS :mutex_lock (&kvm->lock); - kvm->arch.disabled_quirks |= cap->args[0 ] & kvm_caps.supported_quirks; +WRITE_ONCE (kvm->arch.disabled_quirks, + kvm->arch.disabled_quirks | (cap->args[0 ] & kvm_caps.supported_quirks));mutex_unlock (&kvm->lock); r =0 ;break ;
arch/x86/kvm/x86.h +1 -1
@@ -304,7 +304,7 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)static inline bool kvm_check_has_quirk (struct kvm *kvm,u64 quirk) { -return !(kvm->arch.disabled_quirks & quirk); +return !(READ_ONCE (kvm->arch.disabled_quirks) & quirk); } static __always_inline void kvm_request_l1tf_flush_l1d(void)
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 7/18 KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into kvm_arch_flush_shadow_memslot()
In reply to Sean ChristophersonFold kvm_mmu_zap_memslot() into its sole caller so that its GFN range structure can be used to trigger guest_memfd invalidations regardless of whether KVM will do a partial or full zap of the MMU.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -20
@@ -7560,8 +7560,14 @@ static void kvm_mmu_zap_memslot_pages_and_flush(struct kvm *kvm,kvm_mmu_remote_flush_or_zap (kvm, &invalid_list, flush); } -static void kvm_mmu_zap_memslot (struct kvm *kvm, -struct kvm_memory_slot *slot) +static inline bool kvm_memslot_flush_zap_all (struct kvm *kvm) +{ +return kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && +kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL ); +} + +void kvm_arch_flush_shadow_memslot (struct kvm *kvm, +struct kvm_memory_slot *slot) {struct kvm_gfn_range range = { .slot = slot,
@@ -7572,25 +7578,14 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, };bool flush; -write_lock (&kvm->mmu_lock); - flush =kvm_unmap_gfn_range (kvm, &range); -kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); -write_unlock (&kvm->mmu_lock); -} - -static inline bool kvm_memslot_flush_zap_all (struct kvm *kvm) -{ -return kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && -kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL ); -} - -void kvm_arch_flush_shadow_memslot (struct kvm *kvm, -struct kvm_memory_slot *slot) -{ -if (kvm_memslot_flush_zap_all (kvm)) +if (kvm_memslot_flush_zap_all (kvm)) {kvm_mmu_zap_all_fast (kvm); -else -kvm_mmu_zap_memslot (kvm, slot); + }else { +write_lock (&kvm->mmu_lock); + flush =kvm_unmap_gfn_range (kvm, &range); +kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); +write_unlock (&kvm->mmu_lock); + } }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 8/18 KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves
In reply to Sean ChristophersonSplit kvm_mmu_zap_all_fast() into a "front half" and a "back half", where the front half is everything that runs with mmu_lock held for write, and the back half is the code that runs outside of mmu_lock. This will allow putting more code inside kvm_arch_flush_shadow_memslot()'s critical section without having to take mmu_lock twice in quick succession.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +25 -12
@@ -6921,20 +6921,11 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)kvm_mmu_commit_zap_page (kvm, &invalid_list); } -/* -* Fast invalidate all shadow pages and use lock-break technique -* to zap obsolete pages. -* -* It's required when memslot is being deleted or VM is being -* destroyed, in these cases, we should ensure that KVM MMU does -* not use any resource of the being-deleted slot or all slots -* after calling the function. -*/ -static void kvm_mmu_zap_all_fast (struct kvm *kvm) +static void __kvm_mmu_zap_all_fast_front_half (struct kvm *kvm) {lockdep_assert_held (&kvm->slots_lock); +lockdep_assert_held_write (&kvm->mmu_lock); -write_lock (&kvm->mmu_lock);trace_kvm_mmu_zap_all_fast (kvm);/*
@@ -6971,8 +6962,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );kvm_zap_obsolete_pages (kvm); +} -write_unlock (&kvm->mmu_lock); +static void __kvm_mmu_zap_all_fast_back_half (struct kvm *kvm) +{ +lockdep_assert_held (&kvm->slots_lock); +lockdep_assert_not_held (&kvm->mmu_lock);/* * Zap the invalidated TDP MMU roots, all SPTEs must be dropped before
@@ -6986,6 +6981,24 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_tdp_mmu_zap_invalidated_roots (kvm, true); } +/* +* Fast invalidate all shadow pages and use lock-break technique +* to zap obsolete pages. +* +* It's required when memslot is being deleted or VM is being +* destroyed, in these cases, we should ensure that KVM MMU does +* not use any resource of the being-deleted slot or all slots +* after calling the function. +*/ +static void kvm_mmu_zap_all_fast (struct kvm *kvm) +{ +write_lock (&kvm->mmu_lock); +__kvm_mmu_zap_all_fast_front_half (kvm); +write_unlock (&kvm->mmu_lock); + +__kvm_mmu_zap_all_fast_back_half (kvm); +} +int kvm_mmu_init_vm (struct kvm *kvm) {int r, i;
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 9/18 KVM: x86/mmu: Use split "zap all fast" helpers when invalidating memslot
In reply to Sean ChristophersonManually invoke the front half and back half of the "zap all fast" flow when invalidating a memslot so that mmu_lock is acquired at function scope
in kvm_arch_flush_shadow_memslot(). This will allow putting more code
inside the critical section without having to take mmu_lock twice in quick succession.
Opportunistically open code checking whether or not to do the fast zap, to discourage removing the local "zap_all" in a future cleanup, i.e. to ensure the SLOT_ZAP_ALL quirk is queried exactly once. Processing the front half but not the back half of the fast zap (if SLOT_ZAP_ALL were disabled concurrently) would result in KVM unnecessarily keeping invalid TDP MMU roots until the VM is destroyed.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +11 -10
@@ -7573,12 +7573,6 @@ static void kvm_mmu_zap_memslot_pages_and_flush(struct kvm *kvm,kvm_mmu_remote_flush_or_zap (kvm, &invalid_list, flush); } -static inline bool kvm_memslot_flush_zap_all (struct kvm *kvm) -{ -return kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && -kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL ); -} -void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) {
@@ -7589,16 +7583,23 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, .may_block = true, .attr_filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED, }; +bool zap_all = kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && +kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL );bool flush; -if (kvm_memslot_flush_zap_all (kvm)) { -kvm_mmu_zap_all_fast (kvm); +write_lock (&kvm->mmu_lock); + +if (zap_all) { +__kvm_mmu_zap_all_fast_front_half (kvm); }else { -write_lock (&kvm->mmu_lock); flush =kvm_unmap_gfn_range (kvm, &range);kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); -write_unlock (&kvm->mmu_lock); } + +write_unlock (&kvm->mmu_lock); + +if (zap_all) +__kvm_mmu_zap_all_fast_back_half (kvm); }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 10/18 KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
In reply to Sean ChristophersonWire up a gmem_invalidate_range() call for SNP VMs, and use it to force vCPUs to reload/recheck their guest-provided VMSA if the backing gmem page is being invalidated, e.g. is being PUNCH_HOLE'd. Use the same core logic to handle invalidations as VMX does for the APIC-access page, as the two concepts are nearly identical: shove the physical address of a page into the vCPU's control structure:
1. Snapshot the invalidation sequence counter
2. Grab the pfn (from guest_memfd in this case)
3. Acquire mmu_lock for read
4. Re-request reload if retry is needed, otherwise commit the change.
Note, the re-request action in #4 is necessary as KVM's retry logic is fuzzy, i.e. can get false positives. If the guest_memfd page has been dropped, at some point a subsequent reload will fail to get a PFN from guest_memfd, and KVM will fail KVM_RUN. If the retry was due to a false positive, KVM will retry until there are no relevant MMU notifier events (and will retry in the "outer" loop, i.e. will drop locks and resched as needed).
Note #2! Take care to invalidate the VMSA when a relevant memslot is DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated on memslot bindings (KVM doesn't know what GFN range(s) to invalidate without a binding). And more importantly, the VMSA mapping requires a memslot, i.e. must be invalidated if its memslots disappears, regardless of the state of the underlying guest_memfd inode.
Failure to invalidate the vCPU's control.vmsa_pa (which is checked by pre_sev_run()) can prevent KVM from properly freeing the page as firmware will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU is actively running, i.e. if VMSA page is in-use. That in turn leads to an RMP #PF on the next use, as the page will still be assigned to the SNP VM.
SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O Tainted: [U]=USER, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 Call Trace: <TASK> dump_stack_lvl+0x54/0x70 rmpupdate+0x12c/0x140 rmp_make_shared+0x3b/0x60 sev_gmem_invalidate+0xe0/0x170 [kvm_amd] delete_from_page_cache_batch+0x1d8/0x220 truncate_inode_pages_range+0x120/0x3d0 kvm_gmem_fallocate+0x19a/0x270 [kvm] vfs_fallocate+0x1bc/0x1f0 __x64_sys_fallocate+0x48/0x70 do_syscall_64+0x10a/0x480 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x496c7e </TASK> ------------[ cut here ]------------ SEV: Failed to update RMP entry for PFN 0x78d198 error -14 WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345 CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O Tainted: [U]=USER, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd] Call Trace: <TASK> delete_from_page_cache_batch+0x1d8/0x220 truncate_inode_pages_range+0x120/0x3d0 kvm_gmem_fallocate+0x19a/0x270 [kvm] vfs_fallocate+0x1bc/0x1f0 __x64_sys_fallocate+0x48/0x70 do_syscall_64+0x10a/0x480 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x496c7e </TASK> irq event stamp: 20689 hardirqs last enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60 hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60 softirqs last enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140 softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140 ---[ end trace 0000000000000000 ]--- BUG: unable to handle page fault for address: ffff99a64d198000 #PF: supervisor write access in kernel mode #PF: error_code(0x80000003) - RMP violation PGD 13eb001067 P4D 13eb001067 PUD 78d1d1063 PMD 1184e0063 PTE 800000078d198163 SEV-SNP: PFN 0x78d198, RMP entry: [0x6030000000144001 - 0x000000000000000f] Oops: Oops: 0003 [#1] SMP CPU: 3 UID: 0 PID: 31407 Comm: highlanderd_hea Tainted: G U W O Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 RIP: 0010:prep_new_page+0x67/0x220 Call Trace: <TASK> get_page_from_freelist+0x1c40/0x1c70 __alloc_frozen_pages_noprof+0xca/0x1f0 alloc_pages_mpol+0x10b/0x1b0 alloc_pages_noprof+0x81/0x90 pte_alloc_one+0x1b/0xd0 do_pte_missing+0xdf/0x1020 handle_mm_fault+0x7c7/0xb20 do_user_addr_fault+0x268/0x6b0 exc_page_fault+0x67/0xa0 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x4a6b1e </TASK> gsmi: Log Shutdown Reason 0x03 CR2: ffff99a64d198000 ---[ end trace 0000000000000000 ]--- RIP: 0010:prep_new_page+0x67/0x220
Drop the pseudo-TODO comment about needing to pin the page if guest_memfd every supports migration, as integrating with invalidations events means KVM will Just Work if/when page migration is ever supported (assuming SNP hardware supports migrating VMSA pages).
Note #3, invalidate() and invalidate_range() have _completely_ different semantics; the new invalidate_range() is a true invalidation, whereas the existing invalidate() is really a "make shared" operation. Ignore the confusing naming and poor Kconfig bundling for the moment to minimize the delta for LTS kernels, the mess will be cleaned up shortly.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 ++ arch/x86/include/asm/kvm_host.h | 6 +++ arch/x86/kvm/mmu/mmu.c | 5 ++ arch/x86/kvm/svm/sev.c | 80 ++++++++++++++++++++++++++---- arch/x86/kvm/svm/svm.c | 2 + arch/x86/kvm/svm/svm.h | 2 + arch/x86/kvm/x86.c | 6 +++ include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 4 ++ 9 files changed, 99 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +4 -0
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL(reload_vmsa) KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked)
@@ -148,6 +149,9 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) KVM_X86_OP_OPTIONAL(gmem_invalidate) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL(gmem_invalidate_range) +#endif #endif #undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +6 -0
@@ -122,6 +122,8 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_VMSA_PAGE_RELOAD \ + KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \ KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
@@ -1878,6 +1880,7 @@ struct kvm_x86_ops {int (*vm_copy_enc_context_from)(struct kvm *kvm,unsigned int source_fd);int (*vm_move_enc_context_from)(struct kvm *kvm,unsigned int source_fd);void (*guest_memory_reclaimed)(struct kvm *kvm); +void (*reload_vmsa)(struct kvm_vcpu *vcpu);int (*get_feature_msr)(u32 msr,u64 *data);
@@ -1902,6 +1905,9 @@ struct kvm_x86_ops {void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order);void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range); +#endif int (*gmem_max_mapping_level)(struct kvm *kvm,kvm_pfn_t pfn,bool is_private); };
arch/x86/kvm/mmu/mmu.c +5 -0
@@ -7589,6 +7589,11 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,write_lock (&kvm->mmu_lock); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +if (slot->gmem.file) +kvm_arch_gmem_invalidate_range (kvm, &range); +#endif +if (zap_all) {__kvm_mmu_zap_all_fast_front_half (kvm); }else {
arch/x86/kvm/svm/sev.c +69 -11
128 lines not on this page · raw message
arch/x86/kvm/svm/svm.c +2 -0
@@ -5445,12 +5445,14 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .mem_enc_register_region = sev_mem_enc_register_region, .mem_enc_unregister_region = sev_mem_enc_unregister_region, .guest_memory_reclaimed = sev_guest_memory_reclaimed, + .reload_vmsa = sev_snp_reload_vmsa, .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, .gmem_prepare = sev_gmem_prepare, .gmem_invalidate = sev_gmem_invalidate, + .gmem_invalidate_range = sev_gmem_invalidate_range, .gmem_max_mapping_level = sev_gmem_max_mapping_level, #endif .check_emulate_instruction = svm_check_emulate_instruction,
arch/x86/kvm/svm/svm.h +2 -0
@@ -996,6 +996,7 @@ static inline struct page *snp_safe_alloc_page(void) {return snp_safe_alloc_page_node (numa_node_id (),GFP_KERNEL_ACCOUNT ); } +void sev_snp_reload_vmsa (struct kvm_vcpu *vcpu);int sev_vcpu_create (struct kvm_vcpu *vcpu);void sev_free_vcpu (struct kvm_vcpu *vcpu);
@@ -1010,6 +1011,7 @@ extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code);int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order);void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu);void sev_free_decrypted_vmsa (struct kvm_vcpu *vcpu,struct vmcb_save_area *vmsa);
arch/x86/kvm/x86.c +6 -0
@@ -8170,6 +8170,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } +if (kvm_check_request (KVM_REQ_VMSA_PAGE_RELOAD , vcpu)) +kvm_x86_call (reload_vmsa)(vcpu); } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
@@ -10599,6 +10601,10 @@ void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) {kvm_x86_call (gmem_invalidate)(start, end); } +void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_x86_call (gmem_invalidate_range)(kvm, range); +}#endif #endif
include/linux/kvm_host.h +1 -0
@@ -2608,6 +2608,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);#endif #ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY
virt/kvm/guest_memfd.c +4 -0
@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif }if (flush)
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] The guest_memfd invalidation flow fails to synchronously wait for vCPUs to exit guest mode…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] The guest_memfd invalidation flow fails to synchronously wait for vCPUs to exit guest mode before allowing the host to free the VMSA page, resulting in RMPUPDATE FAIL_INUSE hardware errors and a host kernel panic. --
--- Patch [10]: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped --- commit ceb579c141d8889038624fb707ecde51b8427194
Author: Sean Christopherson <[email protected]>
KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
This commit wires up a gmem_invalidate_range() call for SNP VMs, forcing vCPUs to reload their VMSA if the backing guest_memfd page is invalidated (e.g., via PUNCH_HOLE).
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0
[ ... ]
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 194
Quoted 11 lines
virt/kvm/guest_memfd.c +4
-0@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif }if (flush)
[Severity: Critical]
Does this invalidation path need a way to guarantee a synchronous wait before returning?
Since VMSA pages typically do not have TDP MMU SPTEs, kvm_mmu_unmap_gfn_range() will return false, leaving flush as false.
Because flush remains false, the subsequent kvm_flush_remote_tlbs(kvm) call is skipped.
Meanwhile, sev_gmem_invalidate_range() relies on KVM_REQ_VMSA_PAGE_RELOAD and kvm_make_request_and_kick() to send an asynchronous IPI to the vCPU:
arch/x86/kvm/svm/sev.c:sev_gmem_invalidate_range() {
...
kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
...
}Because the IPI is asynchronous and the TLB flush (which would otherwise provide a synchronous wait) is skipped, the host could proceed to free the VMSA page before the vCPU actually drops it and exits guest mode.
Could this lead to the very RMPUPDATE FAIL_INUSE hardware error and host kernel panic that the commit is trying to resolve, if a host userspace process issues fallocate(PUNCH_HOLE) on the guest_memfd region?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Sean Christopherson
maintainerRe: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
Quoted 35 lines · On Thu, Jul 09, 2026, [email protected] wrote:
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0[ ... ]
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 194
Quoted 11 lines
virt/kvm/guest_memfd.c +4
-0@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif }if (flush)[Severity: Critical]
Does this invalidation path need a way to guarantee a synchronous wait before returning?Since VMSA pages typically do not have TDP MMU SPTEs,
kvm_mmu_unmap_gfn_range()will return false, leaving flush as false.Because flush remains false, the subsequent
kvm_flush_remote_tlbs(kvm)call is skipped.Meanwhile,
sev_gmem_invalidate_range()relies onKVM_REQ_VMSA_PAGE_RELOADandkvm_make_request_and_kick()to send an asynchronous IPI to the vCPU:arch/x86/kvm/svm/sev.c:sev_gmem_invalidate_range() { ... kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu); ... }
Quoted 3 lines
Because the IPI is asynchronous and the TLB flush (which would otherwise provide a synchronous wait) is skipped, the host could proceed to free the VMSA page before the vCPU actually drops it and exits guest mode.
No, KVM_REQ_VMSA_PAGE_RELOAD is declared with KVM_REQUEST_WAIT, i.e. KVM will ensure the target CPUs have received the IPI before continuing on. And the presence of the request will prevent them from re-entering the guest with the now-defunct VMSA.
Sean Christopherson
maintainerPATCH v4 11/18 KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid
In reply to Sean ChristophersonTrack the guest as having a guest-provided VMSA as soon as control.vmsa_pa is invalidated, instead of waiting to see if the guest-provided VMSA is usable, so that KVM doesn't switch back to the original VMSA instead of exiting to userspace (due to an invalid VMSA). By the time a vCPU tries to load a guest-provided VMSA, KVM has already communicated "success" for AP creation, i.e. KVM has committed to using the guest-provided VMSA.
--- arch/x86/kvm/svm/sev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
arch/x86/kvm/svm/sev.c +17 -17
@@ -4005,23 +4005,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (gpa)) -return ; - - slot =gfn_to_memslot (vcpu->kvm, gfn); -if (!slot) -return ; - - mmu_seq = kvm->mmu_invalidate_seq; -smp_rmb (); - -/* -* The new VMSA will be private memory guest memory, so retrieve the -* PFN from the gmem backend. -*/ -if (kvm_gmem_get_pfn (vcpu->kvm, slot, gfn, &pfn, &page,NULL )) -return ; -/* * From this point forward, the VMSA will always be a guest-mapped page * rather than the initial one allocated by KVM in svm->sev_es.vmsa. In
@@ -4033,6 +4016,23 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)*/ svm->sev_es.snp_has_guest_vmsa = true; +if (!VALID_PAGE (gpa)) +return ; + + slot =gfn_to_memslot (vcpu->kvm, gfn); +if (!slot) +return ; + + mmu_seq = kvm->mmu_invalidate_seq; +smp_rmb (); + +/* +* The new VMSA will be private memory guest memory, so retrieve the +* PFN from the gmem backend. +*/ +if (kvm_gmem_get_pfn (vcpu->kvm, slot, gfn, &pfn, &page,NULL )) +return ; +read_lock (&kvm->mmu_lock);/* * Save the guest-provided GPA. If retry is needed, then KVM will try
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 12/18 KVM: x86: Guard .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE=y
In reply to Sean ChristophersonWrap the .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE so that non-SEV code doesn't try to wire up a callback without doing the necessary enabling.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 +++- arch/x86/include/asm/kvm_host.h | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +3 -1
@@ -146,12 +146,14 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) -KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) +#endif KVM_X86_OP_OPTIONAL(gmem_invalidate) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE KVM_X86_OP_OPTIONAL(gmem_invalidate_range) #endif +KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) #endif #undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1903,7 +1903,9 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +#endif void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range);
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 13/18 KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
In reply to Sean ChristophersonPass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA can be made optional in the future, i.e. so that guest_memfd can pass INVALID_GPA. This will allow reworking the hook into a general .convert() hook, i.e. so that it can be used for both "make private" and "make shared" operations, at which point guest_memfd may not have a corresponding GFN, e.g. when converting memory back to a shared state when a folio is freed.
Opportunistically swap the ordering of @pfn and @gpa params for kvm_x86_ops.gmem_prepare() to match that of kvm_arch_gmem_prepare().
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/sev.c | 3 ++- arch/x86/kvm/svm/svm.h | 2 +- arch/x86/kvm/x86.c | 4 ++-- include/linux/kvm_host.h | 2 +- virt/kvm/guest_memfd.c | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1904,7 +1904,7 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +int (*gmem_prepare)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order);#endif void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/svm/sev.c +2 -1
@@ -5112,9 +5112,10 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)return false; } -int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order) +int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order) {struct kvm_sev_info *sev =to_kvm_sev_info (kvm); +gfn_t gfn =gpa_to_gfn (gpa);kvm_pfn_t pfn_aligned;gfn_t gfn_aligned;int level, rc;
arch/x86/kvm/svm/svm.h +1 -1
@@ -1009,7 +1009,7 @@ int sev_cpu_init(struct svm_cpu_data *sd);int sev_dev_get_attr (u32 group,u64 attr,u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code); -int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order);void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end);void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);
arch/x86/kvm/x86.c +2 -2
@@ -10590,9 +10590,9 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm) }#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,int max_order) +int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order) { -return kvm_x86_call (gmem_prepare)(kvm, pfn, gfn, max_order); +return kvm_x86_call (gmem_prepare)(kvm, gpa, pfn, max_order); }#endif
include/linux/kvm_host.h +1 -1
@@ -2573,7 +2573,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,#endif /* CONFIG_KVM_GUEST_MEMFD */ #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,int max_order); +int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order);#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +1 -1
@@ -66,7 +66,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE kvm_pfn_t pfn =folio_file_pfn (folio, index);gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff; -int rc =kvm_arch_gmem_prepare (kvm, gfn, pfn,folio_order (folio)); +int rc =kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio));if (rc) {pr_warn_ratelimited ("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n" , index, gfn, pfn, rc);
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 3 lines · Sean Christopherson
Pass a GPA instead of a GFN to
kvm_arch_gmem_prepare()so that the GFN/GPA can be made optional in the future, i.e. so thatguest_memfdcan passINVALID_GPA. This will allow reworking the hook into a general .convert()
How about defining a INVALID_GFN instead of using gpa just so INVALID_GPA can be passed? The SNP side takes a gfn, so it seems like the 2 PAGE_SHIFTs are extra work.
Quoted 12 lines
hook, i.e. so that it can be used for both "make private" and "make shared" operations, at which point
guest_memfdmay not have a corresponding GFN, e.g. when converting memory back to a shared state when a folio is freed.Opportunistically swap the ordering of @pfn and @gpa params for
kvm_x86_ops.gmem_prepare()to match that ofkvm_arch_gmem_prepare().No functional change intended.
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 9 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
Pass a GPA instead of a GFN to
kvm_arch_gmem_prepare()so that the GFN/GPA can be made optional in the future, i.e. so thatguest_memfdcan passINVALID_GPA. This will allow reworking the hook into a general .convert()How about defining a
INVALID_GFNinstead of using gpa just soINVALID_GPAcan be passed? The SNP side takes a gfn, so it seems like the 2PAGE_SHIFTsare extra work.
They are, but I am very hesitant to define INVALID_GFN as I'm worried about the potential for subtle bugs if both INVALID_GPA and INVALID_GFN exist. E.g. I almost made a goof in this exact series where I was going to do:
gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa); ... if (gfn == INVALID_GFN) return;
Where snp_guest_vmsa_gpa is set to INVALID_GPA.
I suppose we could define INVALID_GFN to gpa_to_gfn(INVALID_GPA), but that feels weird, and I feel more comfortable having magic invalid values in only one domain.
Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 15 lines · Sean Christopherson
Quoted 9 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
Pass a GPA instead of a GFN to
kvm_arch_gmem_prepare()so that the GFN/GPA can be made optional in the future, i.e. so thatguest_memfdcan passINVALID_GPA. This will allow reworking the hook into a general .convert()How about defining a
INVALID_GFNinstead of using gpa just soINVALID_GPAcan be passed? The SNP side takes a gfn, so it seems like the 2PAGE_SHIFTsare extra work.They are, but I am very hesitant to define
INVALID_GFNas I'm worried about the potential for subtle bugs if bothINVALID_GPAandINVALID_GFNexist. E.g. I almost made a goof in this exact series where I was going to do:
Quoted 5 lines
gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa); ... if (gfn == INVALID_GFN)
When you did this check INVALID_GFN was defined as (~0) too I suppose...
Quoted 6 lines
return;Where
snp_guest_vmsa_gpais set toINVALID_GPA.I suppose we could define
INVALID_GFNtogpa_to_gfn(INVALID_GPA), but that feels weird, and I feel more comfortable having magic invalid values in only one domain.
Maybe INVALID_GPA and INVALID_GFN should be defined such that both gpa_to_gfn and gfn_to_gpa are correct for both. Isn't the relationship between gpa and gfn always supposed to be always a fixed shift apart?
When a GPA is passed to the arch function, it almost feels like now the arch should be validating PAGE_ALIGNED(gpa).
Mentioned this off-list, another thing that seems incongruent is that one parameter is in the address domain (gpa) and the other is in the page frame domain (pfn).
Sean Christopherson
maintainerRe: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 17 lines · Ackerley Tng
Quoted 15 lines · Sean Christopherson
Quoted 9 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
Pass a GPA instead of a GFN to
kvm_arch_gmem_prepare()so that the GFN/GPA can be made optional in the future, i.e. so thatguest_memfdcan passINVALID_GPA. This will allow reworking the hook into a general .convert()How about defining a
INVALID_GFNinstead of using gpa just soINVALID_GPAcan be passed? The SNP side takes a gfn, so it seems like the 2PAGE_SHIFTsare extra work.They are, but I am very hesitant to define
INVALID_GFNas I'm worried about the potential for subtle bugs if bothINVALID_GPAandINVALID_GFNexist. E.g. I almost made a goof in this exact series where I was going to do:
Quoted 7 lines
Quoted 5 lines
gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa); ... if (gfn == INVALID_GFN)When you did this check
INVALID_GFNwas defined as (~0) too I suppose...
Ya. I didn't actually get that far, because INVALID_GFN didn't exist, but that was my plan.
Quoted 9 lines
Quoted 6 lines
return;Where
snp_guest_vmsa_gpais set toINVALID_GPA.I suppose we could define
INVALID_GFNtogpa_to_gfn(INVALID_GPA), but that feels weird, and I feel more comfortable having magic invalid values in only one domain.Maybe
INVALID_GPAandINVALID_GFNshould be defined such that bothgpa_to_gfnandgfn_to_gpaare correct for both.
Oh, _that_ was actually what I almost broke. I was going to track the gfn instead of the gpa, and then do:
svm->sev_es.snp_guest_vmsa_gfn = INVALID_GFN; ... gpa = gfn_to_gpa(svm->sev_es.snp_guest_vmsa_gfn); if (gpa == INVALID_GPA) return;
That would have failed miserably, because even if if INVALID_GFN is -1ull, the lower bits would be zero after the shift, i.e. wouldn't match INVALID_GPA.
I _think_ make that work, by doing something like:
#define INVALID_GFN ((~(gfn_t)0) >> PAGE_SHIFT) #define INVALID_GPA ((gpa_t)INVALID_GFN << PAGE_SHIFT)
Which I actually don't mind conceptually, but I'm still not sure it's a good idea in practice. For whatever reason, converting between INVALID_GFN and INVALID_GPA just feels wrong.
Quoted 2 lines
Isn't the relationship between gpa and gfn always supposed to be always a fixed shift apart?
For valid GPAs, yes. But if a value is completely bogus, is there really any relationship?
Quoted 2 lines
When a GPA is passed to the arch function, it almost feels like now the arch should be validating
PAGE_ALIGNED(gpa).
Not sure I follow this.
Quoted 3 lines
Mentioned this off-list, another thing that seems incongruent is that one parameter is in the address domain (gpa) and the other is in the page frame domain (pfn).
Yeah, I definitely don't love this either. Hrm.
Aha! Idea! What if we just kinda sorta punt on the INVALID_GFN concept for the moment? This series doesn't actually _need_ to detect an INVALID_GFN, the only "usage" is in a sanity check:
if (to_private) {
if (WARN_ON_ONCE(!kvm || gpa == INVALID_GPA))
return -EIO;
return sev_gmem_make_private(kvm, gpa, pfn, nr_pages, order);
}and that can very easily be this, which is even better from a safety perspective, and is actually somewhat sensible given that it also ensures KVM isn't trying to assign a gfn with the C-bit set. (Untested, i.e. I may botched the math, but the idea is sound).
if (to_private) {
if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SIZE)))
return -EIO;
return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
}Then in kvm_gmem_free_folio(), open code -1ull for the gfn:
WARN_ON_ONCE(kvm_arch_gmem_convert(NULL, -1ull, folio_file_pfn(folio, 0), folio_nr_pages(folio), folio_order(folio), false));
Which is all a little gross? But I think it's less gross than passing a gpa, and we don't risk delaying this and the in-place conversion series trying to find the perfect answer.
Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 54 lines · Sean Christopherson
Quoted 17 lines · Ackerley Tng
Quoted 15 lines · Sean Christopherson
Quoted 9 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
Pass a GPA instead of a GFN to
kvm_arch_gmem_prepare()so that the GFN/GPA can be made optional in the future, i.e. so thatguest_memfdcan passINVALID_GPA. This will allow reworking the hook into a general .convert()How about defining a
INVALID_GFNinstead of using gpa just soINVALID_GPAcan be passed? The SNP side takes a gfn, so it seems like the 2PAGE_SHIFTsare extra work.They are, but I am very hesitant to define
INVALID_GFNas I'm worried about the potential for subtle bugs if bothINVALID_GPAandINVALID_GFNexist. E.g. I almost made a goof in this exact series where I was going to do:Quoted 7 lines
Quoted 5 lines
gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa); ... if (gfn == INVALID_GFN)When you did this check
INVALID_GFNwas defined as (~0) too I suppose...Ya. I didn't actually get that far, because
INVALID_GFNdidn't exist, but that was my plan.Quoted 9 lines
Quoted 6 lines
return;Where
snp_guest_vmsa_gpais set toINVALID_GPA.I suppose we could define
INVALID_GFNtogpa_to_gfn(INVALID_GPA), but that feels weird, and I feel more comfortable having magic invalid values in only one domain.Maybe
INVALID_GPAandINVALID_GFNshould be defined such that bothgpa_to_gfnandgfn_to_gpaare correct for both.Oh, _that_ was actually what I almost broke. I was going to track the gfn instead of the gpa, and then do:
svm->sev_es.snp_guest_vmsa_gfn = INVALID_GFN; ... gpa = gfn_to_gpa(svm->sev_es.snp_guest_vmsa_gfn); if (gpa == INVALID_GPA) return;That would have failed miserably, because even if if
INVALID_GFNis -1ull, the lower bits would be zero after the shift, i.e. wouldn't matchINVALID_GPA.
Quoted 9 lines
I _think_ make that work, by doing something like:
#define INVALID_GFN ((~(gfn_t)0) >> PAGE_SHIFT) #define INVALID_GPA ((gpa_t)INVALID_GFN << PAGE_SHIFT)Which I actually don't mind conceptually, but I'm still not sure it's a good idea in practice. For whatever reason, converting between
INVALID_GFNandINVALID_GPAjust feels wrong.
The above isn't a conversion, it's picking/defining sentinel values that are dependent on each other :)
We could also go all the way where gfn_t is always retrieved with a getter, and the getter always checks for the sentinel value, and then maybe the compiler will optimize out the check sometimes, but this is probably just C's way, where when invalid values are not possible they're not checked (pre-optimized out).
No strong opinions, I do prefer passing gfn over gpa though.
Quoted 11 lines
Quoted 2 lines
Isn't the relationship between gpa and gfn always supposed to be always a fixed shift apart?
For valid GPAs, yes. But if a value is completely bogus, is there really any relationship?
Quoted 2 lines
When a GPA is passed to the arch function, it almost feels like now the arch should be validating
PAGE_ALIGNED(gpa).Not sure I follow this.
If sev_gmem_convert() was handed a gpa, it seems to imply that the caller could have passed in an unaligned gpa, which means sev_gmem_convert() should do
if (WARN_ON_ONCE(!PAGE_ALIGNED(gpa)))
return -EIO;
which is even more extra work in addition to the conversion from gfn to gpa in the caller and back within sev_gmem_convert().
Quoted 31 lines
Quoted 3 lines
Mentioned this off-list, another thing that seems incongruent is that one parameter is in the address domain (gpa) and the other is in the page frame domain (pfn).
Yeah, I definitely don't love this either. Hrm.
Aha! Idea! What if we just kinda sorta punt on the
INVALID_GFNconcept for the moment? This series doesn't actually _need_ to detect anINVALID_GFN, the only "usage" is in a sanity check:if (to_private) { if (WARN_ON_ONCE(!kvm || gpa == INVALID_GPA)) return -EIO; return sev_gmem_make_private(kvm, gpa, pfn, nr_pages, order); }and that can very easily be this, which is even better from a safety perspective, and is actually somewhat sensible given that it also ensures KVM isn't trying to assign a gfn with the C-bit set. (Untested, i.e. I may botched the math, but the idea is sound).
if (to_private) { if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SIZE))) return -EIO; return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order); }Then in
kvm_gmem_free_folio(), open code -1ull for the gfn:
Quoted 8 lines
WARN_ON_ONCE(kvm_arch_gmem_convert(NULL, -1ull, folio_file_pfn(folio, 0), folio_nr_pages(folio), folio_order(folio), false));Which is all a little gross? But I think it's less gross than passing a gpa, and we don't risk delaying this and the in-place conversion series trying to find the perfect answer.
Yes thanks, we should go ahead with the GFN version, we can definitely update this later, if at all.
I have some more questions, though not blocking:
+ I didn't notice that NULL was passed here. Is this kind of leaking SNP details into guest_memfd, that SNP doesn't need to know which kvm it is to convert a page to shared? Or is it that in principle converting to shared means "removing from ANY VM" and hence kvm = NULL?
+ Is the purpose of checking for the bit being set to set up a contract? It's better than passing gpa, but TDX has the opposite definition for the bit, with 1 == shared. TDX doesn't use this callback so that's fine, but it's hard even for SNP to check since IIUC with VTOM 1 == shared? What if we just not check and depend only on kvm == NULL?
How about this: as a lifecycle callback, guest_memfd calling this function should be saying "guest_memfd is doing a conversion of nr_pages beginning at this pfn to shared/private, and by the way here's the kvm and gfn since I have the information." (all arguments presented if available) and SNP says "I'll take action, for to_shared I don't need kvm or gfn" (just ignore kvm and gfn for to_shared conversions)
Sean Christopherson
maintainerRe: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
Quoted 24 lines · Ackerley Tng
Quoted 23 lines · Sean Christopherson
Quoted 2 lines
Maybe
INVALID_GPAandINVALID_GFNshould be defined such that bothgpa_to_gfnandgfn_to_gpaare correct for both.Oh, _that_ was actually what I almost broke. I was going to track the gfn instead of the gpa, and then do:
svm->sev_es.snp_guest_vmsa_gfn = INVALID_GFN; ... gpa = gfn_to_gpa(svm->sev_es.snp_guest_vmsa_gfn); if (gpa == INVALID_GPA) return;That would have failed miserably, because even if if
INVALID_GFNis -1ull, the lower bits would be zero after the shift, i.e. wouldn't matchINVALID_GPA.I _think_ make that work, by doing something like:
#define INVALID_GFN ((~(gfn_t)0) >> PAGE_SHIFT) #define INVALID_GPA ((gpa_t)INVALID_GFN << PAGE_SHIFT)
Quoted 6 lines
Quoted 3 lines
Which I actually don't mind conceptually, but I'm still not sure it's a good idea in practice. For whatever reason, converting between
INVALID_GFNandINVALID_GPAjust feels wrong.The above isn't a conversion, it's picking/defining sentinel values that are dependent on each other :)
My point is that the entire reason the sential values need to be dependent on each other is that if we add INVALID_GFN, then KVM will inevitably convert INVALID_GFN to a gpa and check the result against INVALID_GPA.
Quoted 21 lines
Quoted 9 lines
if (to_private) { if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SIZE))) return -EIO; return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order); }Then in
kvm_gmem_free_folio(), open code -1ull for the gfn:Quoted 8 lines
WARN_ON_ONCE(kvm_arch_gmem_convert(NULL, -1ull, folio_file_pfn(folio, 0), folio_nr_pages(folio), folio_order(folio), false));Which is all a little gross? But I think it's less gross than passing a gpa, and we don't risk delaying this and the in-place conversion series trying to find the perfect answer.
Yes thanks, we should go ahead with the GFN version, we can definitely update this later, if at all.
Quoted 6 lines
I have some more questions, though not blocking:
+ I didn't notice that NULL was passed here. Is this kind of leaking SNP details into
guest_memfd, that SNP doesn't need to know which kvm it is to convert a page to shared? Or is it that in principle converting to shared means "removing from ANY VM" and hence kvm = NULL?
Neither? It's saying "there's no VM at this point", because there's not. And I wouldn't consider that an SNP detail, because any CoCo implementation has to work that way in practice. Requiring an active VM to reclaim memory for host usage would be unusable.
Quoted 1 line
+ Is the purpose of checking for the bit being set to set up a contract?
No, it's to harden against against screwing things up in the futures. And the check isn't against the C-bit, it's against the max legal gfn KVM supports, which just so happens to have implications with respect to the C-bit.
Quoted 4 lines
It's better than passing gpa, but TDX has the opposite definition for the bit, with 1 == shared. TDX doesn't use this callback so that's fine, but it's hard even for SNP to check since IIUC with VTOM 1 == shared? What if we just not check and depend only on kvm == NULL?
Because the to_private path requires a valid gfn
Quoted 6 lines
How about this: as a lifecycle callback,
guest_memfdcalling this function should be saying "guest_memfdis doing a conversion ofnr_pagesbeginning at this pfn to shared/private, and by the way here's the kvm and gfn since I have the information." (all arguments presented if available) and SNP says "I'll take action, forto_sharedI don't need kvm or gfn" (just ignore kvm and gfn forto_sharedconversions)
No? That's more or less what I proposed, except I guess for "by the way, here's the kvm and gfn" for to_shared. Because I do NOT think we should walk all bindings to find the kvm+gfn pairs, and because I think we should firmly establish that to_private requires a kvm+gfn pair, and to_shared does not. As above, in pracitce, that needs to hold true for any CoCo VM.
Sean Christopherson
maintainerPATCH v4 14/18 KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure
In reply to Sean ChristophersonDrop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP alreads logs more precise messages in all error paths, and whether or not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail.
--- virt/kvm/guest_memfd.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
virt/kvm/guest_memfd.c +3 -7
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE kvm_pfn_t pfn =folio_file_pfn (folio, index);gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff; -int rc =kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio)); -if (rc) { -pr_warn_ratelimited ("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n" , - index, gfn, pfn, rc); -return rc; - } -#endif +return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio)); +#else return 0 ; +#endif }/*
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 14/18] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure
Quoted 34 lines · Sean Christopherson
Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP alreads logs more precise messages in all error paths, and whether or not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail.
Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/guest_memfd.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)virt/kvm/guest_memfd.c +3
-8@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE kvm_pfn_t pfn =folio_file_pfn (folio, index);gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff; -int rc =kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio)); -if (rc) { -pr_warn_ratelimited ("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n" , - index, gfn, pfn, rc); -return rc; - } -#endif +return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio)); +#else return 0 ; +#endif }/* --
Quoted 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerPATCH v4 15/18 KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
In reply to Sean ChristophersonFold __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio() and extend the #ifdef guard to cover all of the kvm_gmem_prepare_folio(). Providing a single-use, ~2 line tail-call helper adds no value. In fact, due to having to reconstitue the gfn, the separate helper is very arguably a net negative.
No functional change intended.
--- virt/kvm/guest_memfd.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
virt/kvm/guest_memfd.c +7 -14
@@ -60,19 +60,6 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } -static int __kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot, -pgoff_t index,struct folio *folio) -{ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -kvm_pfn_t pfn =folio_file_pfn (folio, index); -gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff; - -return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), pfn,folio_order (folio)); -#else -return 0 ; -#endif -} -/* * Process @folio, which contains @gfn, so that the guest can use it. * The folio must be locked and the gfn must be contained in @slot.
@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slostatic int kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,struct folio *folio) { +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE pgoff_t index;/*
@@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, index =kvm_gmem_get_index (slot, gfn); index =ALIGN_DOWN (index,folio_nr_pages (folio)); -return __kvm_gmem_prepare_folio (kvm, slot, index, folio); +return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), +folio_file_pfn (folio, index), +folio_order (folio)); +#else +return 0 ; +#endif }/*
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The patch breaks the GPA alignment for huge folios by passing the original, unaligned faulting…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch breaks the GPA alignment for huge folios by passing the original, unaligned faulting gfn to kvm_arch_gmem_prepare(), resulting in a mismatch between the unaligned GPA and the aligned base PFN of the folio. --
--- Patch [15]: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller --- commit 30508b546ec3ffd0655d6bdccee0095661b93a2e
Author: Sean Christopherson <[email protected]>
KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
This commit folds __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio(), extending the #ifdef guard to cover the entirety of the function and eliminating the tail-call helper.
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0
[ ... ]
Quoted 9 lines
virt/kvm/guest_memfd.c +1
-0@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slostatic int kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,struct folio *folio) { +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE pgoff_t index;/*
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 94
Quoted 7 lines
virt/kvm/guest_memfd.c +3
-1@@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, index =kvm_gmem_get_index (slot, gfn); index =ALIGN_DOWN (index,folio_nr_pages (folio)); -return __kvm_gmem_prepare_folio (kvm, slot, index, folio); +return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), +folio_file_pfn (folio, index), +folio_order (folio));
[Severity: Medium]
When folding this helper, did we lose the GFN alignment for huge folios?
In the old __kvm_gmem_prepare_folio() helper, the gfn was recalculated from the aligned index:
gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
By passing the original, unaligned faulting gfn to kvm_arch_gmem_prepare() alongside the aligned base PFN from folio_file_pfn(), does this create a mismatch between the unaligned GPA and the aligned base PFN for huge folios?
While guest_memfd currently restricts allocations to order-0 folios, this would pass an offset GPA with a base PFN to the architecture hooks once huge folios are supported, violating the API contract and potentially corrupting the guest mapping (e.g., if a backend like SEV-SNP falls back to 4K mappings for a huge folio).
Quoted 4 lines
virt/kvm/guest_memfd.c +3
-0+#else +return 0 ; +#endif }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Sean Christopherson
maintainerRe: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
Quoted 18 lines · On Thu, Jul 09, 2026, [email protected] wrote:
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0[ ... ]
Quoted 12 lines
virt/kvm/guest_memfd.c +1
-0@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slostatic int kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,struct folio *folio) { +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE pgoff_t index;/* @@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, index =kvm_gmem_get_index (slot, gfn); index =ALIGN_DOWN (index,folio_nr_pages (folio));
Quoted 7 lines
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 94
Quoted 4 lines
virt/kvm/guest_memfd.c +3
-1-return __kvm_gmem_prepare_folio (kvm, slot, index, folio); +return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), +folio_file_pfn (folio, index), +folio_order (folio));[Severity: Medium]
When folding this helper, did we lose the GFN alignment for huge folios?
Ugh, I totally missed that, obviously. Aligning the index in the caller using a gfn, and then subtly computing a new gfn is just mean. I'll fix this by squashing in "Align the gfn as well as the pfn when "preparing" a folio"
Quoted 22 lines
In the old
__kvm_gmem_prepare_folio()helper, the gfn was recalculated from the aligned index:gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;By passing the original, unaligned faulting gfn to
kvm_arch_gmem_prepare()alongside the aligned base PFN fromfolio_file_pfn(), does this create a mismatch between the unaligned GPA and the aligned base PFN for huge folios?While
guest_memfdcurrently restricts allocations to order-0 folios, this would pass an offset GPA with a base PFN to the architecture hooks once huge folios are supported, violating the API contract and potentially corrupting the guest mapping (e.g., if a backend like SEV-SNP falls back to 4K mappings for a huge folio).Quoted 4 lines
virt/kvm/guest_memfd.c +3
-0+#else +return 0 ; +#endif }Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Sean Christopherson
maintainerPATCH v4 16/18 KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()
In reply to Sean ChristophersonTweak the guest_memfd prepare() hook to explicitly pass the number of pages to "prepare", so that the prepare() and invalidate() hooks can be combined into a single convert() hook, and because the existing code is outright broken if a guest_memfd folio is comprised of more than one page (which can't happen, yet). guest_memfd aligns the pfn down, and the SNP code *tries* to create a corresponding huge entry, but if the RMP must use 4KiB entries for whatever reason, KVM will only "prepare" the first pfn, and not the actual pfn that needs to be converted/prepared.
Alternatively, @max_order could simply be repurposed as _the_ @order, but that will fall apart when in-place conversion comes along, at which point KVM will need to deal with conversions that aren't strictly bound to a one folio.
Note, kvm_gmem_prepare_folio() still mishandles hugepages, as it aligns only the pfn, i.e. would pass the wrong gfn+pfn pair. This will be fixed shortly.
WARN in the SNP code if the number of pages to prepare is anything other than '1', i.e. if guest_memfd is trying to prepare/convert more than a single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion greater than order-0 folios.
Opportunistically convert the existing WARN in the prepare flow to a only fire once, e.g. so that a guest_memfd bug doesn't unintentionally Dos the kernel by spamming the log.
--- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/svm/sev.c | 6 +++++- arch/x86/kvm/svm/svm.h | 3 ++- arch/x86/kvm/x86.c | 5 +++-- include/linux/kvm_host.h | 3 ++- virt/kvm/guest_memfd.c | 7 ++++--- 6 files changed, 18 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
@@ -1904,7 +1904,8 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int (*gmem_prepare)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order); +int (*gmem_prepare)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order);#endif void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/svm/sev.c +5 -1
@@ -5112,7 +5112,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)return false; } -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order) +int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order) {struct kvm_sev_info *sev =to_kvm_sev_info (kvm);gfn_t gfn =gpa_to_gfn (gpa);
@@ -5124,6 +5125,9 @@ int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)if (!sev_snp_guest (kvm))return 0 ; +if (WARN_ON_ONCE (nr_pages !=1 )) +return -EIO ; + rc =snp_lookup_rmpentry (pfn, &assigned, &level);if (rc) { pr_err_ratelimited("SEV: Failed to look up RMP entry: GFN %llx PFN %llx error %d\n",
arch/x86/kvm/svm/svm.h +2 -1
@@ -1009,7 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);int sev_dev_get_attr (u32 group,u64 attr,u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code); -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order); +int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order);void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end);void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);
arch/x86/kvm/x86.c +3 -2
@@ -10590,9 +10590,10 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm) }#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order) +int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order) { -return kvm_x86_call (gmem_prepare)(kvm, gpa, pfn, max_order); +return kvm_x86_call (gmem_prepare)(kvm, gpa, pfn, nr_pages, max_order); }#endif
include/linux/kvm_host.h +2 -1
@@ -2573,7 +2573,8 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,#endif /* CONFIG_KVM_GUEST_MEMFD */ #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn,int max_order); +int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order);#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
virt/kvm/guest_memfd.c +4 -3
@@ -70,6 +70,7 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, struct folio *folio) { #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE + unsigned long nr_pages = folio_nr_pages(folio);pgoff_t index;/*
@@ -85,12 +86,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,* The order will be passed when creating the guest_memfd, and * checked when creating memslots. */ -WARN_ON (!IS_ALIGNED (slot->gmem.pgoff,folio_nr_pages (folio))); +WARN_ON_ONCE (!IS_ALIGNED (slot->gmem.pgoff, nr_pages)); index =kvm_gmem_get_index (slot, gfn); - index =ALIGN_DOWN (index,folio_nr_pages (folio)); + index =ALIGN_DOWN (index, nr_pages);return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), -folio_file_pfn (folio, index), +folio_file_pfn (folio, index), nr_pages,folio_order (folio));#else return 0 ;
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] Data race between guest memory population and fault-driven zeroing leads to host kernel panic via RMP violation. --
--- Patch [16]: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare() --- commit b12c982da03eae2bbf9f63075885746a3653b16b
Author: Sean Christopherson <[email protected]>
KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()
This commit modifies the guest_memfd prepare() hook to explicitly pass the number of pages, allowing prepare() and invalidate() hooks to be combined into a single convert() hook. It also adds a WARN_ON_ONCE to the SNP code if more than a single 4KiB page is prepared.
virt/kvm/guest_memfd.c
Quoted 4 lines
virt/kvm/guest_memfd.c +0
-0
[ ... ]
[Severity: Critical]
This is a pre-existing issue, but could a data race between guest memory population and fault-driven zeroing lead to a host kernel panic?
Looking at __kvm_gmem_populate() in virt/kvm/guest_memfd.c:
folio_unlock(folio);
if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
KVM_MEMORY_ATTRIBUTE_PRIVATE,
KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
ret = -EINVAL;
goto out_put_folio;
}
ret = post_populate(kvm, gfn, pfn, src_page, opaque);
if (!ret)
folio_mark_uptodate(folio);The folio is unlocked before the memory is populated or marked up-to-date. If an unprivileged host userspace process concurrently triggers a page fault on the same private GFN, could kvm_gmem_get_pfn() fetch this unlocked folio?
kvm_gmem_get_pfn() would observe the !uptodate folio and zero it via the host direct map:
if (!folio_test_uptodate(folio)) {
clear_highpage(folio_page(folio, 0));
folio_mark_uptodate(folio);
}If this host zeroing overlaps with or occurs after rmp_make_private() in the population callback, wouldn't this host write cause an RMP violation and crash the host kernel?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Re: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()
In reply to Sean ChristophersonQuoted 23 lines · Sean Christopherson
Tweak the
guest_memfdprepare()hook to explicitly pass the number of pages to "prepare", so that theprepare()andinvalidate()hooks can be combined into a singleconvert()hook, and because the existing code is outright broken if aguest_memfdfolio is comprised of more than one page (which can't happen, yet).guest_memfdaligns the pfn down, and the SNP code *tries* to create a corresponding huge entry, but if the RMP must use 4KiB entries for whatever reason, KVM will only "prepare" the first pfn, and not the actual pfn that needs to be converted/prepared.Alternatively, @max_order could simply be repurposed as _the_ @order, but that will fall apart when in-place conversion comes along, at which point KVM will need to deal with conversions that aren't strictly bound to a one folio.
Note,
kvm_gmem_prepare_folio()still mishandles hugepages, as it aligns only the pfn, i.e. would pass the wrong gfn+pfn pair. This will be fixed shortly.WARN in the SNP code if the number of pages to prepare is anything other than '1', i.e. if
guest_memfdis trying to prepare/convert more than a single 4KiB page, assev_gmem_prepare()doesn't actually handle conversion greater than order-0 folios.
Quoted 9 lines
Opportunistically convert the existing WARN in the prepare flow to a only fire once, e.g. so that a
guest_memfdbug doesn't unintentionally Dos the kernel by spamming the log.Fixes b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")Signed-off-by Sean Christopherson <[email protected]>[...snip...]
Sean Christopherson
maintainerPATCH v4 17/18 KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
In reply to Sean ChristophersonWhen preparing a guest_memfd folio, align the gfn as well as the pfn, i.e. preserve the effective gfn+pfn pair provided by the caller. If the folio is larger than order-0, aligning the pfn but not the gfn would create a mismatch set and unintentionally crash the guest due to assigning the pfn at the wrong gfn.
--- virt/kvm/guest_memfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/guest_memfd.c +1 -1
@@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,* checked when creating memslots. */ WARN_ON_ONCE (!IS_ALIGNED (slot->gmem.pgoff, nr_pages)); + gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); - index =ALIGN_DOWN (index, nr_pages); return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), folio_file_pfn(folio, index), nr_pages,
Signature · 1 line
2.55.0.795.g602f6c329a-goog
[email protected]
AI reviewRe: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Medium] Aligning GFN down without validating memslot boundaries introduces host out-of-bounds physical memory access when preparing huge folios.
Pre-existing issues:
- [Critical] Concurrent execution of KVM_GMEM_POPULATE and vCPU page faults allows unprivileged userspace to trigger host hardware crashes and guest data corruption. --
--- Patch [17]: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio --- commit 5debe95f8712f79807df583e8d5e927a97b4c012
Author: Sean Christopherson <[email protected]>
KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
When preparing a guest_memfd folio, this patch aligns the gfn as well as the pfn to preserve the effective gfn+pfn pair. This prevents a mismatch set and unintentional guest crashes due to assigning the pfn at the wrong gfn for huge folios.
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 91
Quoted 11 lines
virt/kvm/guest_memfd.c +1
-1@@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,* checked when creating memslots. */ WARN_ON_ONCE (!IS_ALIGNED (slot->gmem.pgoff, nr_pages)); + gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); - index =ALIGN_DOWN (index, nr_pages);
[Severity: Medium]
If a userspace caller creates a guest_memfd memslot with a base_gfn that is not aligned to the huge folio size, will computing the index from the aligned gfn result in a misaligned index?
When passed to kvm_arch_gmem_prepare(), couldn't this cause the iteration to go past the end of the huge folio, resulting in an out-of-bounds host physical memory access?
[Severity: Critical]
This is a pre-existing issue, but I noticed a potential race condition between KVM_GMEM_POPULATE and vCPU page faults in this same file.
Looking at __kvm_gmem_populate(), the folio lock is dropped before calling post_populate():
virt/kvm/guest_memfd.c:__kvm_gmem_populate() {
...
folio_unlock(folio);
...
ret = post_populate(kvm, gfn, pfn, src_page, opaque);
...
}Could a concurrent vCPU page fault on the same GFN call kvm_gmem_get_pfn(), acquire the folio lock, see that it is not uptodate, and clear the page while the populate thread is executing post-populate updates like KVM_SEV_SNP_LAUNCH_UPDATE?
virt/kvm/guest_memfd.c:kvm_gmem_get_pfn() {
...
if (!folio_test_uptodate(folio)) {
clear_highpage(folio_page(folio, 0));
...
}Would this allow unprivileged userspace to trigger host hardware crashes (Machine Check Exceptions) and guest data corruption simply by running a vCPU thread concurrently with the population ioctl?
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), folio_file_pfn(folio, index), nr_pages,
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Re: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
Quoted 37 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Medium] Aligning GFN down without validating memslot boundaries introduces host out-of-bounds physical memory access when preparing huge folios.Pre-existing issues:
- [Critical] Concurrent execution ofKVM_GMEM_POPULATEand vCPU page faults allows unprivileged userspace to trigger host hardware crashes and guest data corruption. ----- Patch [17]: [PATCH v4 17/18] KVM:
guest_memfd: Align the gfn as well as the pfn when "preparing" a folio --- commit5debe95f8712f79807df583e8d5e927a97b4c012
Author: Sean Christopherson <[email protected]>KVM:
guest_memfd: Align the gfn as well as the pfn when "preparing" a folioWhen preparing a
guest_memfdfolio, this patch aligns the gfn as well as the pfn to preserve the effective gfn+pfn pair. This prevents a mismatch set and unintentional guest crashes due to assigning the pfn at the wrong gfn for huge folios.virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 91
Quoted 11 lines
virt/kvm/guest_memfd.c +1
-1@@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,* checked when creating memslots. */ WARN_ON_ONCE (!IS_ALIGNED (slot->gmem.pgoff, nr_pages)); + gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); - index =ALIGN_DOWN (index, nr_pages);[Severity: Medium]
If a userspace caller creates aguest_memfdmemslot with abase_gfnthat is not aligned to the huge folio size, will computing the index from the aligned gfn result in a misaligned index?
Quoted 4 lines
When passed to
kvm_arch_gmem_prepare(), couldn't this cause the iteration to go past the end of the huge folio, resulting in an out-of-bounds host physical memory access?
This sounds like it would be a problem. We could perhaps leave this until huge pages are supported?
Quoted 4 lines
[Severity: Critical]
This is a pre-existing issue, but I noticed a potential race condition betweenKVM_GMEM_POPULATEand vCPU page faults in this same file.
We're good, vCPU page faults can't happen until the VM is finalized, populate must be done before finalizing.
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
Quoted 18 lines · Ackerley Tng
Quoted 17 lines · [email protected] writes:
virt/kvm/guest_memfd.c static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · line 91
Quoted 11 lines
virt/kvm/guest_memfd.c +1
-1@@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,* checked when creating memslots. */ WARN_ON_ONCE (!IS_ALIGNED (slot->gmem.pgoff, nr_pages)); + gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); - index =ALIGN_DOWN (index, nr_pages);[Severity: Medium]
If a userspace caller creates aguest_memfdmemslot with abase_gfnthat is not aligned to the huge folio size, will computing the index from the aligned gfn result in a misaligned index?
Quoted 7 lines
Quoted 4 lines
When passed to
kvm_arch_gmem_prepare(), couldn't this cause the iteration to go past the end of the huge folio, resulting in an out-of-bounds host physical memory access?This sounds like it would be a problem. We could perhaps leave this until huge pages are supported?
Yeah, for sure.
Sean Christopherson
maintainerPATCH v4 18/18 KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
In reply to Sean ChristophersonSmush guest_memfd's prepare() and invalidate() hooks into a common convert() flow, as they are effectively two sides of the same coin: they're invoked when private/shared memory is about to made accessible visible to the guest/host. I.e. prepare() is really "make private", and invalidate() is really "make shared".
Using a single API fixes the wart where invalidate() isn't actually doing invalidation, avoids adding yet another gmem Kconfig, avoids having to come up with a name to describe the "make shared" behavior, and will yield more intuitive code when in-place conversion comes along.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 5 ++--- arch/x86/include/asm/kvm_host.h | 7 +++---- arch/x86/kvm/Kconfig | 2 +- arch/x86/kvm/svm/sev.c | 21 ++++++++++++++++++--- arch/x86/kvm/svm/svm.c | 3 +-- arch/x86/kvm/svm/svm.h | 5 ++--- arch/x86/kvm/x86.c | 15 ++++----------- include/linux/kvm_host.h | 7 +++---- virt/kvm/Kconfig | 2 +- virt/kvm/guest_memfd.c | 25 ++++++++++++------------- 10 files changed, 47 insertions(+), 45 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -3
@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +KVM_X86_OP_OPTIONAL_RET0(gmem_convert) #endif -KVM_X86_OP_OPTIONAL(gmem_invalidate) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE KVM_X86_OP_OPTIONAL(gmem_invalidate_range)#endif
arch/x86/include/asm/kvm_host.h +3 -4
@@ -1903,11 +1903,10 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int (*gmem_prepare)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int (*gmem_convert)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);#endif -void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range);#endif
arch/x86/kvm/Kconfig +1 -1
@@ -160,7 +160,7 @@ config KVM_AMD_SEV depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM select KVM_GENERIC_MEMORY_ATTRIBUTES - select HAVE_KVM_ARCH_GMEM_PREPARE + select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_INVALIDATE select HAVE_KVM_ARCH_GMEM_POPULATE help
arch/x86/kvm/svm/sev.c +18 -3
@@ -5112,8 +5112,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)return false; } -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order) +static int sev_gmem_make_private (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order) {struct kvm_sev_info *sev =to_kvm_sev_info (kvm);gfn_t gfn =gpa_to_gfn (gpa);
@@ -5164,8 +5164,9 @@ int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,return 0 ; } -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +static void sev_gmem_make_shared (kvm_pfn_t start,kvm_pfn_t nr_pages) { +kvm_pfn_t end = start + nr_pages;kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))
@@ -5227,6 +5228,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) } } +int sev_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int order,bool to_private) +{ +if (to_private) { +if (WARN_ON_ONCE (!kvm || gpa ==INVALID_GPA )) +return -EIO ; + +return sev_gmem_make_private (kvm, gpa, pfn, nr_pages, order); + } + +sev_gmem_make_shared (pfn, nr_pages); +return 0 ; +} +void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) {struct kvm_vcpu *vcpu;
arch/x86/kvm/svm/svm.c +1 -2
@@ -5450,8 +5450,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, - .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, + .gmem_convert = sev_gmem_convert, .gmem_invalidate_range = sev_gmem_invalidate_range, .gmem_max_mapping_level = sev_gmem_max_mapping_level,#endif
arch/x86/kvm/svm/svm.h +2 -3
@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);int sev_dev_get_attr (u32 group,u64 attr,u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code); -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +int sev_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +4 -11
@@ -10589,18 +10589,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)return !kvm_arch_has_private_mem (kvm); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private) { -return kvm_x86_call (gmem_prepare)(kvm, gpa, pfn, nr_pages, max_order); -} -#endif - -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) -{ -kvm_x86_call (gmem_invalidate)(start, end); +return kvm_x86_call (gmem_convert)(kvm, gpa, pfn, nr_pages, max_order, to_private); }void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) {
include/linux/kvm_host.h +3 -4
@@ -2572,9 +2572,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, }#endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
@@ -2608,7 +2608,6 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end);void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);#endif
virt/kvm/Kconfig +1 -1
@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD select XARRAY_MULTI bool -config HAVE_KVM_ARCH_GMEM_PREPARE +config HAVE_KVM_ARCH_GMEM_CONVERT bool depends on KVM_GUEST_MEMFD
virt/kvm/guest_memfd.c +12 -13
@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)* On successful return the guest sees a zero page so as to avoid * leaking host data and the up-to-date flag is set. */ -static int kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,struct folio *folio) +static int kvm_gmem_make_private (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,struct folio *folio) { -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT unsigned long nr_pages =folio_nr_pages (folio);pgoff_t index;
@@ -90,9 +90,9 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); -return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), +return kvm_arch_gmem_convert (kvm,gfn_to_gpa (gfn),folio_file_pfn (folio, index), nr_pages, -folio_order (folio)); +folio_order (folio), true);#else return 0 ;#endif
@@ -517,14 +517,13 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folreturn MF_DELAYED ; } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT static void kvm_gmem_free_folio (struct folio *folio) { -struct page *page =folio_page (folio,0 ); -kvm_pfn_t pfn =page_to_pfn (page); -int order =folio_order (folio); - -kvm_arch_gmem_invalidate (pfn, pfn + (1ul << order)); +WARN_ON_ONCE (kvm_arch_gmem_convert (NULL ,INVALID_GPA , +folio_file_pfn (folio,0 ), +folio_nr_pages (folio), +folio_order (folio), false)); }#endif
@@ -532,7 +531,7 @@ static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT .free_folio = kvm_gmem_free_folio,#endif };
@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } - r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); + r =kvm_gmem_make_private (kvm, slot, gfn, folio);folio_unlock (folio);
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
virt/kvm/guest_memfd.c static const struct address_space_operations kvm_gmem_aops = { · line 535
Quoted 10 lines · Sean Christopherson
[...snip...]
virt/kvm/guest_memfd.c +1
-1@@ -532,7 +531,7 @@ static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT .free_folio = kvm_gmem_free_folio,
This part looks odd to have CONVERT gate free_folio, maybe this is where it's nice to have CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ifdef the contents of free_folio above.
Not a huge deal, thanks for this cleanup!
Quoted 13 lines
virt/kvm/guest_memfd.c +1
-2#endif };@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } - r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); + r =kvm_gmem_make_private (kvm, slot, gfn, folio);folio_unlock (folio); --2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
In reply to Sean ChristophersonQuoted 6 lines · Sean Christopherson
Smush guest_memfd's
prepare()andinvalidate()hooks into a commonconvert()flow, as they are effectively two sides of the same coin: they're invoked when private/shared memory is about to made accessible visible to the guest/host. I.e.prepare()is really "make private", andinvalidate()is really "make shared".
That holds for SNP, where both paths end in rmp_make_shared(). It doesn't hold for pKVM: conversion has to preserve the page (same physical page, no encryption, which is the point of in-place conversion for us), while teardown has to scrub it, so that a malicious host can't read a newly reclaimed guest page.
Quoted 101 lines
Using a single API fixes the wart where
invalidate()isn't actually doing invalidation, avoids adding yet another gmemKconfig, avoids having to come up with a name to describe the "make shared" behavior, and will yield more intuitive code when in-place conversion comes along.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm-x86-ops.h | 5 ++--- arch/x86/include/asm/kvm_host.h | 7 +++---- arch/x86/kvm/Kconfig | 2 +- arch/x86/kvm/svm/sev.c | 21 ++++++++++++++++++--- arch/x86/kvm/svm/svm.c | 3 +-- arch/x86/kvm/svm/svm.h | 5 ++--- arch/x86/kvm/x86.c | 15 ++++----------- include/linux/kvm_host.h | 7 +++---- virt/kvm/Kconfig | 2 +- virt/kvm/guest_memfd.c | 25 ++++++++++++------------- 10 files changed, 47 insertions(+), 45 deletions(-)arch/x86/include/asm/kvm-x86-ops.h +2
-3@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +KVM_X86_OP_OPTIONAL_RET0(gmem_convert) #endif -KVM_X86_OP_OPTIONAL(gmem_invalidate) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE KVM_X86_OP_OPTIONAL(gmem_invalidate_range)#endif arch/x86/include/asm/kvm_host.h +3
-4@@ -1903,11 +1903,10 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int (*gmem_prepare)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int (*gmem_convert)(struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);#endif -void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range);#endif arch/x86/kvm/Kconfig +1
-1@@ -160,7 +160,7 @@ config KVM_AMD_SEV depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM select KVM_GENERIC_MEMORY_ATTRIBUTES - select HAVE_KVM_ARCH_GMEM_PREPARE + select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_INVALIDATE select HAVE_KVM_ARCH_GMEM_POPULATE helparch/x86/kvm/svm/sev.c +5
-3@@ -5112,8 +5112,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)return false; } -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order) +static int sev_gmem_make_private (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order) {struct kvm_sev_info *sev =to_kvm_sev_info (kvm);gfn_t gfn =gpa_to_gfn (gpa);@@ -5164,8 +5164,9 @@ int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,return 0 ; } -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +static void sev_gmem_make_shared (kvm_pfn_t start,kvm_pfn_t nr_pages) { +kvm_pfn_t end = start + nr_pages;kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))@@ -5227,6 +5228,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) } } +int sev_gmem_convert(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
arch/x86/kvm/svm/sev.c void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) · line 5241
Quoted 10 lines
arch/x86/kvm/svm/sev.c +10
-0+ kvm_pfn_t nr_pages, int order, bool to_private) +{ + if (to_private) { + if (WARN_ON_ONCE(!kvm || gpa == INVALID_GPA)) + return -EIO; + +return sev_gmem_make_private (kvm, gpa, pfn, nr_pages, order); + } + +sev_gmem_make_shared (pfn, nr_pages);
free_folio is the only to_private=false caller today, but the cover letter sketches the other one, a private->shared conversion through kvm_gmem_make_shared(inode, start, end). Once that lands both reach the hook with to_private=false, and as far as I can tell, the only thing separating them is kvm == NULL / INVALID_GPA. That isn't documented as a contract, and nothing in-tree reads it on that branch (the kvm/gpa check is under if (to_private), and sev_gmem_make_shared() doesn't take a kvm at all), so nothing would catch it regressing. The sketch is also inode-scoped, so I'm not sure the conversion path passes a non-NULL kvm at all; if it doesn't, the two aren't distinguishable.
Quoted 132 lines
arch/x86/kvm/svm/sev.c +3
-0+return 0 ; +} +void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) {struct kvm_vcpu *vcpu;arch/x86/kvm/svm/svm.c +1
-2@@ -5450,8 +5450,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, - .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, + .gmem_convert = sev_gmem_convert, .gmem_invalidate_range = sev_gmem_invalidate_range, .gmem_max_mapping_level = sev_gmem_max_mapping_level,#endif arch/x86/kvm/svm/svm.h +2
-3@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);int sev_dev_get_attr (u32 group,u64 attr,u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code); -int sev_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +int sev_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu);arch/x86/kvm/x86.c +4
-11@@ -10589,18 +10589,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)return !kvm_arch_has_private_mem (kvm); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private) { -return kvm_x86_call (gmem_prepare)(kvm, gpa, pfn, nr_pages, max_order); -} -#endif - -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) -{ -kvm_x86_call (gmem_invalidate)(start, end); +return kvm_x86_call (gmem_convert)(kvm, gpa, pfn, nr_pages, max_order, to_private); }void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) {include/linux/kvm_host.h +3
-4@@ -2572,9 +2572,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, }#endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, -kvm_pfn_t nr_pages,int max_order); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_convert (struct kvm *kvm,gpa_t gpa,kvm_pfn_t pfn, +kvm_pfn_t nr_pages,int max_order,bool to_private);#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE @@ -2608,7 +2608,6 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end);void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);#endif virt/kvm/Kconfig +1
-1@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD select XARRAY_MULTI bool -config HAVE_KVM_ARCH_GMEM_PREPARE +config HAVE_KVM_ARCH_GMEM_CONVERT bool depends on KVM_GUEST_MEMFDvirt/kvm/guest_memfd.c +5
-5@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)* On successful return the guest sees a zero page so as to avoid * leaking host data and the up-to-date flag is set. */ -static int kvm_gmem_prepare_folio (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,struct folio *folio) +static int kvm_gmem_make_private (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,struct folio *folio) { -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT unsigned long nr_pages =folio_nr_pages (folio);pgoff_t index;@@ -90,9 +90,9 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, gfn =ALIGN_DOWN (gfn, nr_pages); index =kvm_gmem_get_index (slot, gfn); -return kvm_arch_gmem_prepare (kvm,gfn_to_gpa (gfn), +return kvm_arch_gmem_convert (kvm,gfn_to_gpa (gfn),folio_file_pfn (folio, index), nr_pages, -folio_order (folio)); +folio_order (folio), true);#else return 0 ;#endif
virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 521
Quoted 6 lines
virt/kvm/guest_memfd.c +1
-1@@ -517,14 +517,13 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folreturn MF_DELAYED ; } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT static void kvm_gmem_free_folio (struct folio *folio)
The Kconfig now couples them too: an arch can't take teardown without also taking conversion. pKVM wants teardown and not conversion, so that's a problem for us today, not only once the second caller lands.
This is the same ground as v8 15/46, which ended with the two split under separate Kconfigs. Suzuki confirmed CCA disables both; pKVM wants teardown only, since our conversions are guest-initiated and EL2 has already done the transition by the time the host runs the generic path [1].
I'd rather keep the free path on its own callback. If you want a single hook, passing the caller explicitly instead of leaving it implied by kvm would work too. Merging the two conversion directions is a genuine cleanup, it's the free path I'd like to keep separate.
Thanks,
/fuad
[1] https://lore.kernel.org/all/CA+EHjTwLRzrq0vY0-OosENJ2Gkknerux53aVBj0zGRQUkRQP=A@mail.gmail.com/
Quoted 34 lines
virt/kvm/guest_memfd.c +6
-8{ -struct page *page =folio_page (folio,0 ); -kvm_pfn_t pfn =page_to_pfn (page); -int order =folio_order (folio); - -kvm_arch_gmem_invalidate (pfn, pfn + (1ul << order)); +WARN_ON_ONCE (kvm_arch_gmem_convert (NULL ,INVALID_GPA , +folio_file_pfn (folio,0 ), +folio_nr_pages (folio), +folio_order (folio), false)); }#endif @@ -532,7 +531,7 @@ static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT .free_folio = kvm_gmem_free_folio,#endif };@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } - r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); + r =kvm_gmem_make_private (kvm, slot, gfn, folio);folio_unlock (folio); --2.55.0.795.g602f6c329a-goog
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 3 lines · Fuad Tabba
[...snip...]
Thanks for bringing [1] up, I didn't make the link when this change was being discussed.
pKVM definitely needs a callback when the folio is released from guest_memfd's ownership.
Quoted 6 lines
free_foliois the only to_private=false caller today, but the cover letter sketches the other one, aprivate->sharedconversion throughkvm_gmem_make_shared(inode, start, end). Once that lands both reach the hook with to_private=false, and as far as I can tell, the only thing separating them is kvm == NULL /INVALID_GPA. That isn't
On the folio teardown path kvm has to be non-NULL so that SNP can find out which VM to unassign from, so you're right that INVALID_GPA won't be sufficient to separate conversion to private and teardown, for pKVM, from the arch perspective.
Quoted 20 lines
documented as a contract, and nothing in-tree reads it on that branch (the kvm/gpa check is under if (
to_private), andsev_gmem_make_shared()doesn't take a kvm at all), so nothing would catch it regressing. The sketch is also inode-scoped, so I'm not sure the conversion path passes a non-NULL kvm at all; if it doesn't, the two aren't distinguishable.Quoted 3 lines
[...snip...]
The
Kconfignow couples them too: an arch can't take teardown without also taking conversion. pKVM wants teardown and not conversion, so that's a problem for us today, not only once the second caller lands.This is the same ground as v8 15/46, which ended with the two split under separate Kconfigs. Suzuki confirmed CCA disables both; pKVM wants teardown only, since our conversions are guest-initiated and EL2 has already done the transition by the time the host runs the generic path [1].
Quoted 5 lines
I'd rather keep the free path on its own callback. If you want a single hook, passing the caller explicitly instead of leaving it implied by kvm would work too. Merging the two conversion directions is a genuine cleanup, it's the free path I'd like to keep separate.
Is it okay if later, we add a .gmem_teardown callback with a new CONFIG flag that pKVM can use?
If that's okay, this merge (of function names and of the patch series) can proceed. It's also nicer that the new callback and CONFIG are explicitly (re?)-introduced with pKVM support so the introduction is chronologically linked with pKVM support too :)
Quoted 8 lines
Thanks,
/fuad[1] https://lore.kernel.org/all/CA+EHjTwLRzrq0vY0-OosENJ2Gkknerux53aVBj0zGRQUkRQP=A@mail.gmail.com/
Quoted 3 lines
[...snip...]
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 45 lines · Ackerley Tng
Quoted 3 lines · Fuad Tabba
[...snip...]
Thanks for bringing [1] up, I didn't make the link when this change was being discussed.
pKVM definitely needs a callback when the folio is released from guest_memfd's ownership.
Quoted 6 lines
free_foliois the only to_private=false caller today, but the cover letter sketches the other one, aprivate->sharedconversion throughkvm_gmem_make_shared(inode, start, end). Once that lands both reach the hook with to_private=false, and as far as I can tell, the only thing separating them is kvm == NULL /INVALID_GPA. That isn'tOn the folio teardown path kvm has to be non-NULL so that SNP can find out which VM to unassign from, so you're right that
INVALID_GPAwon't be sufficient to separate conversion to private and teardown, for pKVM, from the arch perspective.Quoted 20 lines
documented as a contract, and nothing in-tree reads it on that branch (the kvm/gpa check is under if (
to_private), andsev_gmem_make_shared()doesn't take a kvm at all), so nothing would catch it regressing. The sketch is also inode-scoped, so I'm not sure the conversion path passes a non-NULL kvm at all; if it doesn't, the two aren't distinguishable.Quoted 3 lines
[...snip...]
The
Kconfignow couples them too: an arch can't take teardown without also taking conversion. pKVM wants teardown and not conversion, so that's a problem for us today, not only once the second caller lands.This is the same ground as v8 15/46, which ended with the two split under separate Kconfigs. Suzuki confirmed CCA disables both; pKVM wants teardown only, since our conversions are guest-initiated and EL2 has already done the transition by the time the host runs the generic path [1].
Quoted 8 lines
Quoted 5 lines
I'd rather keep the free path on its own callback. If you want a single hook, passing the caller explicitly instead of leaving it implied by kvm would work too. Merging the two conversion directions is a genuine cleanup, it's the free path I'd like to keep separate.
Is it okay if later, we add a .gmem_teardown callback with a new CONFIG flag that pKVM can use?
That works for me, and I agree it's nicer to introduce the callback along with its first user.
Sean, is that the plan you have in mind as well? Two things that matter for pKVM when it lands: the new CONFIG needs to be independent of the conversion one, so that free_folio fires for an arch that selects only teardown (pKVM wants teardown and not conversion), and the teardown callback stays distinct from a private->shared conversion. Good point on teardown needing a non-NULL kvm for SNP, incidentally, that's another reason keying off the sentinel wouldn't have worked.
If that's where this is heading, I'm happy.
Cheers,
/fuad
Quoted 14 lines
If that's okay, this merge (of function names and of the patch series) can proceed. It's also nicer that the new callback and CONFIG are explicitly (re?)-introduced with pKVM support so the introduction is chronologically linked with pKVM support too :)
Quoted 8 lines
Thanks,
/fuad[1] https://lore.kernel.org/all/CA+EHjTwLRzrq0vY0-OosENJ2Gkknerux53aVBj0zGRQUkRQP=A@mail.gmail.com/
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 58 lines · Fuad Tabba
Quoted 45 lines · Ackerley Tng
Quoted 3 lines · Fuad Tabba
[...snip...]
Thanks for bringing [1] up, I didn't make the link when this change was being discussed.
pKVM definitely needs a callback when the folio is released from guest_memfd's ownership.
Quoted 6 lines
free_foliois the only to_private=false caller today, but the cover letter sketches the other one, aprivate->sharedconversion throughkvm_gmem_make_shared(inode, start, end). Once that lands both reach the hook with to_private=false, and as far as I can tell, the only thing separating them is kvm == NULL /INVALID_GPA. That isn'tOn the folio teardown path kvm has to be non-NULL so that SNP can find out which VM to unassign from, so you're right that
INVALID_GPAwon't be sufficient to separate conversion to private and teardown, for pKVM, from the arch perspective.Quoted 20 lines
documented as a contract, and nothing in-tree reads it on that branch (the kvm/gpa check is under if (
to_private), andsev_gmem_make_shared()doesn't take a kvm at all), so nothing would catch it regressing. The sketch is also inode-scoped, so I'm not sure the conversion path passes a non-NULL kvm at all; if it doesn't, the two aren't distinguishable.Quoted 3 lines
[...snip...]
The
Kconfignow couples them too: an arch can't take teardown without also taking conversion. pKVM wants teardown and not conversion, so that's a problem for us today, not only once the second caller lands.This is the same ground as v8 15/46, which ended with the two split under separate Kconfigs. Suzuki confirmed CCA disables both; pKVM wants teardown only, since our conversions are guest-initiated and EL2 has already done the transition by the time the host runs the generic path [1].
Quoted 8 lines
Quoted 5 lines
I'd rather keep the free path on its own callback. If you want a single hook, passing the caller explicitly instead of leaving it implied by kvm would work too. Merging the two conversion directions is a genuine cleanup, it's the free path I'd like to keep separate.
Is it okay if later, we add a .gmem_teardown callback with a new CONFIG flag that pKVM can use?
That works for me, and I agree it's nicer to introduce the callback along with its first user.
Quoted 6 lines
Sean, is that the plan you have in mind as well? Two things that matter for pKVM when it lands: the new CONFIG needs to be independent of the conversion one, so that
free_foliofires for an arch that selects only teardown (pKVM wants teardown and not conversion), and the teardown callback stays distinct from aprivate->sharedconversion. Good point on teardown needing a non-NULL kvm for SNP,
Converting back to SHARED doesn't require a non-NULL @kvm pointer. SNP needs the ASID and thus the VM when assigning memory to a VM, but converting memory back to SHARED, i.e. returning to the host, doesn't need that information.
Quoted 2 lines
incidentally, that's another reason keying off the sentinel wouldn't have worked.
It can work, but it would be all kinds of gross, and arguably wrong. To pass a non-NULL @kvm for the true convert-to-shared (not reclaim), guest_memfd would need to iterate over all bindings to get at the @kvm instances. That's "fine", but wasteful and confusing because the SNP conversion is a platform-level thing (as above, unassigning from a VM doesn't require any information about the VM).
Quoted 1 line
If that's where this is heading, I'm happy.
Yeah, this is roughly what the total diff will look like.
x86/include/asm/kvm-x86-ops.h +1 -1
@@ -146,7 +146,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM ) KVM_X86_OP_OPTIONAL_RET0(gmem_convert)#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
x86/include/asm/kvm_host.h +1 -1
@@ -1724,7 +1724,7 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM )int (*gmem_convert)(struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,kvm_pfn_t nr_pages,int max_order,bool to_private);#endif
x86/kvm/Kconfig +1 -0
@@ -161,6 +161,7 @@ config KVM_AMD_SEV select ARCH_HAS_CC_PLATFORM select KVM_GENERIC_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT + select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE select HAVE_KVM_ARCH_GMEM_POPULATE help
x86/kvm/x86.c +10 -0
@@ -10625,6 +10625,16 @@ int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, {return kvm_x86_call (gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, to_private); } +#endif + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +void kvm_arch_gmem_reclaim (kvm_pfn_t pfn,kvm_pfn_t nr_pages,int max_order) +{ +WARN_ON_ONCE (kvm_x86_call (gmem_convert)(NULL ,-1ull , pfn, nr_pages, max_order, false)); +} +#endif + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_x86_call (gmem_invalidate_range)(kvm, range);
linux/kvm_host.h +4 -0
@@ -2614,6 +2614,10 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, kvm_gmem_populate_cb post_populate, void *opaque);#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +void kvm_arch_gmem_reclaim (kvm_pfn_t pfn,kvm_pfn_t nr_pages,int order); +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);#endif
kvm/Kconfig +4 -0
@@ -111,6 +111,10 @@ config HAVE_KVM_ARCH_GMEM_CONVERT bool depends on KVM_GUEST_MEMFD +config HAVE_KVM_ARCH_GMEM_RECLAIM + bool + depends on KVM_GUEST_MEMFD + config HAVE_KVM_ARCH_GMEM_INVALIDATE bool depends on KVM_GUEST_MEMFD
kvm/guest_memfd.c +4 -6
@@ -516,13 +516,11 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folreturn MF_DELAYED ; } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM static void kvm_gmem_free_folio (struct folio *folio) { -WARN_ON_ONCE (kvm_arch_gmem_convert (NULL ,-1ull , -folio_file_pfn (folio,0 ), -folio_nr_pages (folio), -folio_order (folio), false)); +kvm_arch_gmem_reclaim (folio_file_pfn (folio,0 ),folio_nr_pages (folio), +folio_order (folio)); }#endif
@@ -530,7 +528,7 @@ static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM .free_folio = kvm_gmem_free_folio,#endif };
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
In reply to Sean ChristophersonSorry, I just noticed this patch.
Though it may be too late, I have some questions that have been puzzling me for a while :)
Quoted 10 lines · Sean Christopherson
Smush guest_memfd's
prepare()andinvalidate()hooks into a commonconvert()flow, as they are effectively two sides of the same coin: they're invoked when private/shared memory is about to made accessible visible to the guest/host. I.e.prepare()is really "make private", andinvalidate()is really "make shared".Using a single API fixes the wart where
invalidate()isn't actually doing invalidation, avoids adding yet another gmemKconfig, avoids having to come up with a name to describe the "make shared" behavior, and will yield more intuitive code when in-place conversion comes along.
This is reasonable for SEV-SNP, and there is no functional change.
virt/kvm/guest_memfd.c int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 804
Quoted 6 lines
virt/kvm/guest_memfd.c +1
-1@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } - r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); + r =kvm_gmem_make_private (kvm, slot, gfn, folio);
What puzzles me is that kvm_gmem_get_pfn() can be invoked to fault in pages for gmem-only slots in normal VMs, where kvm_arch_has_private_mem() == false.
Would it be odd to invoke .gmem_convert() with to_private being true in such VMs? Or do you plan to add something similar to the following in the future?
if (kvm_gmem_is_private_mem(inode, index)) r = kvm_gmem_make_private(kvm, slot, gfn, folio); else r = kvm_gmem_make_shared(kvm, slot, gfn, folio);
Otherwise, would it look asymmetric with just:
if (kvm_gmem_is_private_mem(inode, index))
r = kvm_gmem_make_private(kvm, slot, gfn, folio);
in kvm_gmem_get_pfn()?
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).
Additionally, would the .gmem_convert() to shared in punch hole be duplicated with the one in kvm_gmem_free_folio() when CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM is true?
So, I'm writing to understand your considerations behind this for our future direction.
Quoted 6 lines
virt/kvm/guest_memfd.c +0
-0folio_unlock (folio);Signature · 1 line
2.55.0.795.g602f6c329a-goog
Sean Christopherson
maintainerRe: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 16 lines · Yan Zhao
virt/kvm/guest_memfd.c int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 804
Quoted 6 lines
virt/kvm/guest_memfd.c +1
-1@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,folio_mark_uptodate (folio); } - r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); + r =kvm_gmem_make_private (kvm, slot, gfn, folio);What puzzles me is that
kvm_gmem_get_pfn()can be invoked to fault in pages for gmem-only slots in normal VMs, wherekvm_arch_has_private_mem()== false.Would it be odd to invoke .gmem_convert() with
to_privatebeing true in such VMs? Or do you plan to add something similar to the following in the future?if (kvm_gmem_is_private_mem(inode, index)) r = kvm_gmem_make_private(kvm, slot, gfn, folio); else r = kvm_gmem_make_shared(kvm, slot, gfn, folio);
Quoted 4 lines
Otherwise, would it look asymmetric with just:
if (kvm_gmem_is_private_mem(inode, index))
r =kvm_gmem_make_private(kvm, slot, gfn, folio);
inkvm_gmem_get_pfn()?
Heh, I should have read this sooner. With help from Xiaoyao and Ackerley, I eventually got to the same place.
https://lore.kernel.org/all/[email protected]
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).
Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
kvm_pfn_t nr_pages, int max_order)
{
return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order);
}
int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
bool to_private)
{
kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
return 0;
}
#endif
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
{
kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
}
#endifRe: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 22 lines · Sean Christopherson
Quoted 16 lines · Yan Zhao
Review int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · line 804
Quoted 6 lines
(quoted diff) +1
-1@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, folio_mark_uptodate(folio); } - r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio); + r = kvm_gmem_make_private(kvm, slot, gfn, folio);What puzzles me is that
kvm_gmem_get_pfn()can be invoked to fault in pages for gmem-only slots in normal VMs, wherekvm_arch_has_private_mem()== false.Would it be odd to invoke .gmem_convert() with
to_privatebeing true in such VMs? Or do you plan to add something similar to the following in the future?if (kvm_gmem_is_private_mem(inode, index)) r = kvm_gmem_make_private(kvm, slot, gfn, folio); else r = kvm_gmem_make_shared(kvm, slot, gfn, folio);Quoted 4 lines
Otherwise, would it look asymmetric with just:
if (kvm_gmem_is_private_mem(inode, index))
r =kvm_gmem_make_private(kvm, slot, gfn, folio);
inkvm_gmem_get_pfn()?
Quoted 4 lines
Heh, I should have read this sooner. With help from Xiaoyao and Ackerley, I eventually got to the same place.
Thanks for the link.
Let me just reply here for easier follow-up of the concern regarding TDX huge pages, which I think is not addressed by the code below.
Quoted 10 lines
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
kvm_pfn_t nr_pages, int max_order)
{
return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order);
}
int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order,
bool to_private)
{
kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
return 0;
}For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
__kvm_gmem_set_attributes(to shared)
|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
But TDX needs kvm pointer, and splitting pages may fail.
Quoted 8 lines
#endif
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
{
kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);
}
#endifIs this kvm_arch_gmem_reclaim() invoked by kvm_gmem_free_folio(), and should TDX not define CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM?
As in [1], for TDX huge pages, you suggested pretending a to-shared conversion in kvm_gmem_punch_hole(). In that case, should we provide a new CONFIG_xxx to prevent it from being invoked by SNP?
@@ -253,13 +294,18 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
kvm_gmem_invalidate_begin(inode, start, end);
- truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+ /*
+ * For demonstration purposes, pretend this is a private=>shared conversion.
+ */
+ r = kvm_gmem_convert(inode, start, end, false);
+ if (!r)
+ truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
kvm_gmem_invalidate_end(inode, start, end);
filemap_invalidate_unlock(inode->i_mapping);
- return 0;
+ return r;
}
[1] https://lore.kernel.org/all/[email protected]/
Or would the following approach acceptable to you ? It renames .gmem_convert() to .gmem_prezap() and invokes it before each kvm_gmem_zap(), so TDX can hook it to perform page splitting before the actual zaps on private pages. Per my understanding, this op servers a different purpose from .gmem_make_private()/.gmem_make_shared() in this patch.
commit 96bf42273e12297fb2192d16add160b169d76157
Author: Sean Christopherson <[email protected]> Date: Fri Jun 26 15:41:24 2026 +0800
KVM: guest_memfd: Add pre-zap arch hook
Add a gmem "pre-zap" hook to allow arch code to take action before a zap, e.g., for shared<=>private conversion, and just as importantly, to let arch code reject/fail a zap, e.g. if the conversion requires new page tables and KVM hits in OOM situation.
The arch code will be used by TDX to split hugepages as necessary to avoid overzapping PTEs, which for all intents and purposes corrupts guest data for TDX VMs (memory is wiped when private PTEs are removed).
---
rebased to gmem in-place conversion v8
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -149,6 +149,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) KVM_X86_OP_OPTIONAL(gmem_invalidate) +KVM_X86_OP_OPTIONAL_RET0(gmem_prezap)#endif #undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +2 -0
@@ -2010,6 +2010,8 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +int (*gmem_prezap)(struct kvm *kvm,gfn_t start,gfn_t end, +enum kvm_gfn_range_filter attr_filter);void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end);int (*gmem_max_mapping_level)(struct kvm *kvm,kvm_pfn_t pfn,bool is_private); };
arch/x86/kvm/Kconfig +1 -0
@@ -147,6 +147,7 @@ config KVM_INTEL_TDX default y depends on INTEL_TDX_HOST select HAVE_KVM_ARCH_GMEM_POPULATE + select HAVE_KVM_ARCH_GMEM_PREZAP help Provides support for launching Intel Trust Domain Extensions (TDX) confidential VMs on Intel processors.
arch/x86/kvm/x86.c +7 -0
@@ -14176,6 +14176,13 @@ void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)kvm_x86_call (gmem_invalidate)(start, end); }#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREZAP +int kvm_arch_gmem_prezap (struct kvm *kvm,gfn_t start,gfn_t end, +enum kvm_gfn_range_filter attr_filter) +{ +return kvm_x86_call (gmem_prezap)(kvm, start, end, attr_filter); +} +#endif #endif int kvm_spec_ctrl_test_value (u64 value)
include/linux/kvm_host.h +5 -0
@@ -2624,6 +2624,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, kvm_gmem_populate_cb post_populate, void *opaque);#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREZAP +int kvm_arch_gmem_prezap (struct kvm *kvm,gfn_t start,gfn_t end, +enum kvm_gfn_range_filter attr_filter); +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end);#endif
include/linux/kvm_types.h +1 -1
@@ -49,7 +49,7 @@ struct kvm_vcpu_init;struct kvm_memslots ;enum kvm_mr_change ; - +enum kvm_gfn_range_filter ;/* * Address types: *
virt/kvm/Kconfig +4 -0
@@ -116,3 +116,7 @@ config HAVE_KVM_ARCH_GMEM_INVALIDATE config HAVE_KVM_ARCH_GMEM_POPULATE bool depends on KVM_GUEST_MEMFD + +config HAVE_KVM_ARCH_GMEM_PREZAP + bool + depends on KVM_GUEST_MEMFD
virt/kvm/guest_memfd.c +54 -1
86 lines not on this page · raw message
Sean Christopherson
maintainerRe: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 23 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order); } int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order, bool to_private) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); return 0; }
Quoted 8 lines
For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
__kvm_gmem_set_attributes(to shared)|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);But TDX needs kvm pointer, and splitting pages may fail.
Ya, but those are very solvable problems. They just don't need to be addressed today, because SNP is the only user of the conversion APIs.
Quoted 10 lines
Quoted 8 lines
#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); } #endifIs this
kvm_arch_gmem_reclaim()invoked bykvm_gmem_free_folio(), and should TDX not defineCONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM?
Correct.
Quoted 3 lines
As in [1], for TDX huge pages, you suggested pretending a to-shared conversion in
kvm_gmem_punch_hole(). In that case, should we provide a newCONFIG_xxxto prevent it from being invoked by SNP?
No? That code was purely for demonstration purpose, I there was zero intent to ever land it. The patch was tagged *** DO NOT MERGE *** for a reason :-)
Quoted 21 lines
virt/kvm/guest_memfd.c +7
-2@@ -253,13 +294,18 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)kvm_gmem_invalidate_begin (inode, start, end); -truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 ); +/* +* For demonstration purposes, pretend this is a private=>shared conversion. +*/ + r =kvm_gmem_convert (inode, start, end, false); +if (!r) +truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 );kvm_gmem_invalidate_end (inode, start, end);filemap_invalidate_unlock (inode->i_mapping); -return 0 ; +return r; }
Quoted 5 lines
Or would the following approach acceptable to you ? It renames .gmem_convert() to .gmem_prezap() and invokes it before each
kvm_gmem_zap(), so TDX can hook it to perform page splitting before the actual zaps on private pages. Per my understanding, this op servers a different purpose from .gmem_make_private()/.gmem_make_shared() in this patch.
Isn't that just kvm_arch_gmem_invalidate_range()? Which was added to fix the SNP VMSA mess. The only thing that's missing is graceful handling of failure.
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order); } int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order, bool to_private) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); return 0; }For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
Quoted 8 lines
Quoted 5 lines
__kvm_gmem_set_attributes(to shared)|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);But TDX needs kvm pointer, and splitting pages may fail.
Ya, but those are very solvable problems. They just don't need to be addressed today, because SNP is the only user of the conversion APIs.
Ok. I'm ok with the change for today's usages. My concern is regarding future TDX huge page support, as I am currently preparing TDX huge page v4. :)
Sorry for the confusion -- I should have stated my intention more clearly.
Previously, for TDX huge pages, you suggested introducing .gmem_convert() to trigger splitting before zapping S-EPT.
With this new direction, should TDX huge pages instead leverage the .gmem_make_shared() op for that purpose?
If so, should we introduce a CONFIG_HAVE_KVM_ARCH_GMEM_PREZAP guard around the .gmem_make_shared() invocation to serve TDX's splitting purpose, in order to keep the two use cases (SNP and TDX) clearly separated?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); } #endif
Quoted 4 lines
Quoted 2 lines
Is this
kvm_arch_gmem_reclaim()invoked bykvm_gmem_free_folio(), and should TDX not defineCONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM?Correct.
If TDX also provides a .gmem_make_shared() callback for the pre-zap (splitting) step during private-to-shared conversions, my concern is that TDX could be inadvertently affected if CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM is unintentionally selected.
Should we provide a way to prevent TDX from accidentally enabling CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM?
Quoted 6 lines
Quoted 3 lines
As in [1], for TDX huge pages, you suggested pretending a to-shared conversion in
kvm_gmem_punch_hole(). In that case, should we provide a newCONFIG_xxxto prevent it from being invoked by SNP?No? That code was purely for demonstration purpose, I there was zero intent to ever land it. The patch was tagged *** DO NOT MERGE *** for a reason :-)
Understood, I noted that it is tagged as *** DO NOT MERGE ***.
However, for TDX huge page v4, should I continue tagging the patch as *** DO NOT MERGE *** and keep it in the series? The reason I ask is that patch [2], which immediately follows the *** DO NOT MERGE *** patch, depends on it, as patch [2] adds the following line in kvm_arch_gmem_convert():
return kvm_x86_call(gmem_convert)(kvm, start, end, to_private);
[2]https://lore.kernel.org/all/[email protected]/
Quoted 21 lines
Quoted 21 lines
virt/kvm/guest_memfd.c +7
-2@@ -253,13 +294,18 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)kvm_gmem_invalidate_begin (inode, start, end); -truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 ); +/* +* For demonstration purposes, pretend this is a private=>shared conversion. +*/ + r =kvm_gmem_convert (inode, start, end, false); +if (!r) +truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 );kvm_gmem_invalidate_end (inode, start, end);filemap_invalidate_unlock (inode->i_mapping); -return 0 ; +return r; }
Quoted 8 lines
Quoted 5 lines
Or would the following approach acceptable to you ? It renames .gmem_convert() to .gmem_prezap() and invokes it before each
kvm_gmem_zap(), so TDX can hook it to perform page splitting before the actual zaps on private pages. Per my understanding, this op servers a different purpose from .gmem_make_private()/.gmem_make_shared() in this patch.Isn't that just
kvm_arch_gmem_invalidate_range()? Which was added to fix the SNP VMSA mess. The only thing that's missing is graceful handling of failure.
Hmm, if .gmem_make_shared() will be used by both SNP and TDX in the future, then it will be invoked:
- in __kvm_gmem_set_attributes() before kvm_gmem_zap() (which can fail) for TDX.
- in __kvm_gmem_set_attributes() after kvm_gmem_zap() (where can't fail) for SNP.
- in kvm_gmem_punch_hole() before kvm_gmem_zap() for TDX.
- in kvm_gmem_free_folio() for SNP.
Since both TDX and SNP would provide callbacks for the .gmem_make_shared() op but invoke it at different points with different failure semantics, we need to provide different CONFIGs and prevent them from being inadvertently selected by an unintended user.
So for simplicity, could I just rename .gmem_convert() to .gmem_prezap() (instead of to .gmem_make_shared()) for TDX in TDX huge page v4?
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 41 lines · Yan Zhao
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order); } int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order, bool to_private) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); return 0; }For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
Quoted 8 lines
Quoted 5 lines
__kvm_gmem_set_attributes(to shared)|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);But TDX needs kvm pointer, and splitting pages may fail.
Ya, but those are very solvable problems. They just don't need to be addressed today, because SNP is the only user of the conversion APIs.
Ok. I'm ok with the change for today's usages. My concern is regarding future TDX huge page support, as I am currently preparing TDX huge page v4. :)
Sorry for the confusion -- I should have stated my intention more clearly.
Quoted 9 lines
Previously, for TDX huge pages, you suggested introducing .gmem_convert() to trigger splitting before zapping S-EPT.
With this new direction, should TDX huge pages instead leverage the .gmem_make_shared() op for that purpose?If so, should we introduce a
CONFIG_HAVE_KVM_ARCH_GMEM_PREZAPguard around the .gmem_make_shared() invocation to serve TDX's splitting purpose, in order to keep the two use cases (SNP and TDX) clearly separated?
Another thing we need to figure out is the ordering... You mentioned that the S-EPT splitting can fail, and I remember you were suggesting that we merge the S-EPTs back on error, something like that?
We'd have to figure out either a point-of-no-return, or some rollback mechanism anyway, so that kind of depends on conversions and the guest_memfd hugepage series. Probably good to defer this discussion to later, when we have more certainty of how it looks like. :)
Quoted 3 lines
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 53 lines · Ackerley Tng
Quoted 41 lines · Yan Zhao
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order); } int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order, bool to_private) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); return 0; }For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
Quoted 8 lines
Quoted 5 lines
__kvm_gmem_set_attributes(to shared)|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);But TDX needs kvm pointer, and splitting pages may fail.
Ya, but those are very solvable problems. They just don't need to be addressed today, because SNP is the only user of the conversion APIs.
Ok. I'm ok with the change for today's usages. My concern is regarding future TDX huge page support, as I am currently preparing TDX huge page v4. :)
Sorry for the confusion -- I should have stated my intention more clearly.
Quoted 9 lines
Previously, for TDX huge pages, you suggested introducing .gmem_convert() to trigger splitting before zapping S-EPT.
With this new direction, should TDX huge pages instead leverage the .gmem_make_shared() op for that purpose?If so, should we introduce a
CONFIG_HAVE_KVM_ARCH_GMEM_PREZAPguard around the .gmem_make_shared() invocation to serve TDX's splitting purpose, in order to keep the two use cases (SNP and TDX) clearly separated?
Quoted 8 lines
Another thing we need to figure out is the ordering... You mentioned that the S-EPT splitting can fail, and I remember you were suggesting that we merge the S-EPTs back on error, something like that?
We'd have to figure out either a point-of-no-return, or some rollback mechanism anyway, so that kind of depends on conversions and the
guest_memfdhugepage series. Probably good to defer this discussion to later, when we have more certainty of how it looks like. :)
+1. Even with my series, I'm getting a wee bit too far into trying to predict the future.
Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()
Quoted 54 lines · Sean Christopherson
Quoted 53 lines · Ackerley Tng
Quoted 50 lines · Yan Zhao
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Quoted 5 lines
Asking this also because there is a .gmem_convert() for TDX huge pages [1]. In [1], .gmem_convert() is invoked to emulate a to-shared conversion in
kvm_gmem_punch_hole(). However, the per-gmem memory attribute for the range to convert may not be shared after the punch hole. Is it acceptable? (To me, the .gmem_convert() in [1] behaves more like .gmem_prezap()).Ya, these concerns got raised by others. pKVM on arm64 in particular wants to hook reclaim but not conversion. The plan is to keep the reclaim and end up with this implementation for x86:
Quoted 12 lines
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order) { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages, max_order); } int kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order, bool to_private) { kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order); return 0; }For TDX huge pages, if we want to trigger private huge page splitting before converting to shared, should we invoke the hooks like this?
Quoted 8 lines
Quoted 5 lines
__kvm_gmem_set_attributes(to shared)|->kvm_arch_gmem_make_shared |->kvm_x86_call(gmem_make_shared)(pfn, nr_pages, max_order);But TDX needs kvm pointer, and splitting pages may fail.
Ya, but those are very solvable problems. They just don't need to be addressed today, because SNP is the only user of the conversion APIs.
Ok. I'm ok with the change for today's usages. My concern is regarding future TDX huge page support, as I am currently preparing TDX huge page v4. :)
Sorry for the confusion -- I should have stated my intention more clearly.
Previously, for TDX huge pages, you suggested introducing .gmem_convert() to trigger splitting before zapping S-EPT.
With this new direction, should TDX huge pages instead leverage the .gmem_make_shared() op for that purpose?If so, should we introduce a
CONFIG_HAVE_KVM_ARCH_GMEM_PREZAPguard around the .gmem_make_shared() invocation to serve TDX's splitting purpose, in order to keep the two use cases (SNP and TDX) clearly separated?
Quoted 5 lines
Quoted 5 lines
Another thing we need to figure out is the ordering... You mentioned that the S-EPT splitting can fail, and I remember you were suggesting that we merge the S-EPTs back on error, something like that?
We'd have to figure out either a point-of-no-return, or some rollback
I think you were referring to the previous issue in hugetlb-based gmem, where
(a) splitting the backend huge folios was performed before (b) zapping secondary MMU mappings. I previously addressed the issue in patch [1] by performing (b) first, and then invoking S-EPT splitting before (b).
While Vishal noted in [3] that performing (a) before (b) could prevent unnecessary zapping of private mappings, S-EPT splitting (the prezap phase) is still better placed before (a) to avoid the need to rollback and merge backend folios in the error path, given that S-EPT splitting may fail due to memory allocation failure.
This is why I think splitting S-EPT deserves a separate .gmem_prezap() callback rather than reusing .gmem_make_shared().
[1] https://github.com/intel-staging/tdx/commit/a8aedac2df44e29247773db3444bc65f7100daa1
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/CAGtprH-eEUzHDUB0CK2V162HHqvE8kT3bAacb6d3xDYJPwBiYA@mail.gmail.com/
Quoted 6 lines
Quoted 3 lines
mechanism anyway, so that kind of depends on conversions and the
guest_memfdhugepage series. Probably good to defer this discussion to later, when we have more certainty of how it looks like. :)+1. Even with my series, I'm getting a wee bit too far into trying to predict the future.
Ok. Then I'll first have the TDX huge pages register to .gmem_prezap() (instead of .gmem_convert() or .gmem_make_shared()).
We can discuss the naming later.
Sean Christopherson
maintainerQuoted 8 lines · Sean Christopherson
Rework KVM's handling of guest-provided (and always guest_memfd-backed) VMSAs to forcefully reclaim VMSA pages when the pages are being freed from their backing gmem instance, e.g. in response to
PUNCH_HOLE. In the worst case scenario, marking the page SHARED in the RMP will fail due to the page beingIN_USE, ultimately leading to RMP #PF violations due toguest_memfdfreeing the memory back to the kernel while it's still assigned to a VM.[...]
Applied 1-12 to kvm-x86 coco. I'll send a v5 for the remaining changes.
[01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
https://github.com/kvm-x86/linux/commit/42a39ad5d592 [02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper https://github.com/kvm-x86/linux/commit/0060569e4f18 [03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable https://github.com/kvm-x86/linux/commit/98ade8c48c28 [04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y https://github.com/kvm-x86/linux/commit/01a96ff30dde [05/18] KVM: x86: Serialize writes to disabled_quirks using kvm->lock https://github.com/kvm-x86/linux/commit/ba76b23ed36a [06/18] KVM: x86: Ensure runtime reads of disabled_quirks are resolved once https://github.com/kvm-x86/linux/commit/ed15cb219992 [07/18] KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into kvm_arch_flush_shadow_memslot() https://github.com/kvm-x86/linux/commit/06d38eaa78fd [08/18] KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves https://github.com/kvm-x86/linux/commit/b27622c4eeb1 [09/18] KVM: x86/mmu: Use split "zap all fast" helpers when invalidating memslot https://github.com/kvm-x86/linux/commit/db095727ff57 [10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped https://github.com/kvm-x86/linux/commit/d1a3c2162334 [11/18] KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid https://github.com/kvm-x86/linux/commit/98fb11aed12b [12/18] KVM: x86: Guard .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE=y https://github.com/kvm-x86/linux/commit/bfbf5fa51145