KVM: SEV: Fix RMP #PF due freeing in-use VMSA
Subsystems (4)
Sean Christopherson
maintainerv2 cover KVM: SEV: Fix RMP #PF due freeing in-use VMSA
Rework KVM's handling of guest-provided (and always guest_memfd-backed) VMSAs to forcefully reclaim VMSA pages when the pages are being freed from their backing gmem instance, e.g. in response to PUNCH_HOLE. In the worst case scenario, marking the page SHARED in the RMP will fail due to the page being IN_USE, ultimately leading to RMP #PF violations due to guest_memfd freeing the memory back to the kernel while it's still assigned to a VM.
Note, the implementation nearly identical to that used by KVM for VMX's APIC access page (which isn't guest controlled, but is migratable and whose PA is shoved directly into a vCPU control structure).
v2:
- Invalidate VMSAs if the memslot is DELETED or MOVED. [Sashiko]
- Limit stable@ patches without a Fixes to 6.12+.
v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (9):
KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP
guests
KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
KVM: Rework .gmem_invalidate() into .gmem_free_folio()
KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into
kvm_arch_flush_shadow_memslot()
KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back"
halves
KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is
zapped
KVM: x86: Guard .gmem_prepare() declarations with
HAVE_KVM_GMEM_PREPARE=y
KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid
arch/x86/include/asm/kvm-x86-ops.h | 8 +- arch/x86/include/asm/kvm_host.h | 10 +- arch/x86/kvm/mmu/mmu.c | 76 +++++++++------ arch/x86/kvm/svm/sev.c | 152 +++++++++++++++++++++-------- arch/x86/kvm/svm/svm.c | 6 +- arch/x86/kvm/svm/svm.h | 8 +- arch/x86/kvm/x86.c | 10 +- include/linux/kvm_host.h | 3 +- virt/kvm/guest_memfd.c | 17 +--- 9 files changed, 197 insertions(+), 93 deletions(-)
base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 1/9 KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
Track the GPA of the guest-provided VMSA used after AP_CREATION events when running SNP guests, instead of simply tracking whether or not the vCPU is using a guest-provided VMSA. KVM needs to know the GPA of the VMSA that's actively being used so that it can react to MMU invalidation events, i.e. so that KVM can drop the VMSA if its backing guest_memfd page is punched out of existence.
Opportunistically rename snp_vmsa_gpa to clarify that it tracks the pending VMSA GPA, whereas snp_guest_vmsa_gpa now tracks the in-use VMSA GPA.
Note! Take care to track the GPA, not the GFN, as VALID_PAGE() won't behave correctly if an invalid GFN is converted to a GPA for checking.
Note #2! Keep snp_has_guest_vmsa so that switching to a guest-provided VMSA is sticky, even if the guest-provided VMSA becomes invalid.
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 14 +++++++++----- arch/x86/kvm/svm/svm.h | 3 ++- 2 files changed, 11 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/sev.c +9 -5
@@ -4003,6 +4003,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)/* Clear use of the VMSA */ svm->vmcb->control.vmsa_pa =INVALID_PAGE ; + svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ;/* * When replacing the VMSA during SEV-SNP AP creation,
@@ -4010,11 +4011,11 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (svm->sev_es.snp_vmsa_gpa)) +if (!VALID_PAGE (svm->sev_es.snp_pending_vmsa_gpa))return ; - gfn =gpa_to_gfn (svm->sev_es.snp_vmsa_gpa); - svm->sev_es.snp_vmsa_gpa =INVALID_PAGE ; + gfn =gpa_to_gfn (svm->sev_es.snp_pending_vmsa_gpa); + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; slot =gfn_to_memslot (vcpu->kvm, gfn);if (!slot)
@@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ + svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);/* Mark the vCPU as runnable */
@@ -4105,10 +4107,10 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm)return -EINVAL ; } - target_svm->sev_es.snp_vmsa_gpa = svm->vmcb->control.exit_info_2; + target_svm->sev_es.snp_pending_vmsa_gpa = svm->vmcb->control.exit_info_2;break ;case SVM_VMGEXIT_AP_DESTROY : - target_svm->sev_es.snp_vmsa_gpa =INVALID_PAGE ; + target_svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ;break ;default : vcpu_unimpl(vcpu, "vmgexit: invalid AP creation request [%#x] from guest\n",
@@ -4791,6 +4793,8 @@ int sev_vcpu_create(struct kvm_vcpu *vcpu)return -ENOMEM ; svm->sev_es.vmsa =page_address (vmsa_page); + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; + svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ; vcpu->arch.guest_tsc_protected =snp_is_secure_tsc_enabled (vcpu->kvm);
arch/x86/kvm/svm/svm.h +2 -1
@@ -271,7 +271,8 @@ struct vcpu_sev_es_state {u64 ghcb_registered_gpa;struct mutex snp_vmsa_mutex;/* Used to handle concurrent updates of VMSA. */ -gpa_t snp_vmsa_gpa; +gpa_t snp_pending_vmsa_gpa; +gpa_t snp_guest_vmsa_gpa;bool snp_ap_waiting_for_reset;bool snp_has_guest_vmsa; };
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 2/9 KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
In reply to Sean ChristophersonExtract the loading/retrieval of a guest-provided VMSA to a separate helper so that KVM can reuse the core logic when refreshing the VMSA after an MMU invalidation from guest_memfd.
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 52 +++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 21 deletions(-)
arch/x86/kvm/svm/sev.c +31 -21
@@ -3979,29 +3979,17 @@ static int snp_begin_psc(struct vcpu_svm *svm)return snp_do_psc (svm); } -/* -* Invoked as part of svm_vcpu_reset() processing of an init event. -*/ -static void sev_snp_init_protected_guest_state (struct kvm_vcpu *vcpu) +static void sev_snp_reload_vmsa (struct kvm_vcpu *vcpu,gpa_t gpa) {struct vcpu_svm *svm =to_svm (vcpu);struct kvm_memory_slot *slot; +gfn_t gfn =gpa_to_gfn (gpa);struct page *page;kvm_pfn_t pfn; -gfn_t gfn; -guard (mutex)(&svm->sev_es.snp_vmsa_mutex); +lockdep_assert_held (&svm->sev_es.snp_vmsa_mutex); -if (!svm->sev_es.snp_ap_waiting_for_reset) -return ; - - svm->sev_es.snp_ap_waiting_for_reset = false; - -/* Mark the vCPU as offline and not runnable */ - vcpu->arch.pv.pv_unhalted = false; -kvm_set_mp_state (vcpu,KVM_MP_STATE_HALTED ); - -/* Clear use of the VMSA */ +/* Clear use of the VMSA. */ svm->vmcb->control.vmsa_pa =INVALID_PAGE ; svm->sev_es.snp_guest_vmsa_gpa =INVALID_PAGE ;
@@ -4011,12 +3999,9 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (svm->sev_es.snp_pending_vmsa_gpa)) +if (!VALID_PAGE (gpa))return ; - gfn =gpa_to_gfn (svm->sev_es.snp_pending_vmsa_gpa); - svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; - slot =gfn_to_memslot (vcpu->kvm, gfn);if (!slot)return ;
@@ -4040,7 +4025,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_has_guest_vmsa = true;/* Use the new VMSA */ - svm->sev_es.snp_guest_vmsa_gpa =gfn_to_gpa (gfn); + svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn);/* Mark the vCPU as runnable */
@@ -4054,6 +4039,31 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)kvm_release_page_clean (page); } +/* +* Invoked as part of svm_vcpu_reset() processing of an init event. +*/ +static void sev_snp_init_protected_guest_state (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); +gpa_t gpa; + +guard (mutex)(&svm->sev_es.snp_vmsa_mutex); + +if (!svm->sev_es.snp_ap_waiting_for_reset) +return ; + + svm->sev_es.snp_ap_waiting_for_reset = false; + +/* Mark the vCPU as offline and not runnable */ + vcpu->arch.pv.pv_unhalted = false; +kvm_set_mp_state (vcpu,KVM_MP_STATE_HALTED ); + + gpa = svm->sev_es.snp_pending_vmsa_gpa; + svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ; + +sev_snp_reload_vmsa (vcpu, gpa); +} +static int sev_snp_ap_creation (struct vcpu_svm *svm) {struct kvm_sev_info *sev =to_kvm_sev_info (svm->vcpu.kvm);
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 3/9 KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
In reply to Sean ChristophersonAlways mark the vCPU as RUNNABLE after responding to AP_CREATE, even if the guest-specified VMSA is unusable, e.g. isn't backed by a memslot or doesn't have a backing guest_memfd page. If the VMSA is unusable, leaving the vCPU in a non-running state will effectively hang the vCPU instead of reporting an error to userspace. This will also allow retrying the VMSA load in the future, to fix a bug where KVM doesn't honor guest_memfd invalidation events, e.g. if AP_CREATION races with PUNCH_HOLE.
--- arch/x86/kvm/svm/sev.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/sev.c +9 -3
@@ -4028,9 +4028,6 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) svm->sev_es.snp_guest_vmsa_gpa = gpa; svm->vmcb->control.vmsa_pa =pfn_to_hpa (pfn); -/* Mark the vCPU as runnable */ -kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE ); -/* * gmem pages aren't currently migratable, but if this ever changes * then care should be taken to ensure svm->sev_es.vmsa is pinned
@@ -4062,6 +4059,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) svm->sev_es.snp_pending_vmsa_gpa =INVALID_PAGE ;sev_snp_reload_vmsa (vcpu, gpa); + +/* +* Mark the vCPU as runnable for CREATE requests, indicated by a valid +* VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will +* fail instead of blocking indefinitely and hanging the vCPU, e.g. if +* the backing guest_memfd page is unavailable. +*/ +if (VALID_PAGE (gpa)) +kvm_set_mp_state (vcpu,KVM_MP_STATE_RUNNABLE ); }static int sev_snp_ap_creation (struct vcpu_svm *svm)
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 4/9 KVM: Rework .gmem_invalidate() into .gmem_free_folio()
In reply to Sean ChristophersonRename .gmem_invalidate() to .gmem_free_folio() as the hook is called when a folio is freed, which is far too late and lack sufficient information for KVM to actually invalidate its usage of the memory. Drop guest_memfd's trampoline and just wire up .free_folio() directly to the arch callback so that the chain of events is clear and obvious.
Opportunistically guard kvm_x86_ops.gmem_free_folio with an ifdef to ensure the callback will actually be called, e.g. so that non-SEV code doesn't try to wire up a callback without enabling CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 +++- arch/x86/include/asm/kvm_host.h | 4 +++- arch/x86/kvm/svm/sev.c | 4 +++- arch/x86/kvm/svm/svm.c | 4 +++- arch/x86/kvm/svm/svm.h | 3 +-- arch/x86/kvm/x86.c | 4 ++-- include/linux/kvm_host.h | 2 +- virt/kvm/guest_memfd.c | 13 +------------ 8 files changed, 17 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +3 -1
@@ -147,7 +147,9 @@ KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) -KVM_X86_OP_OPTIONAL(gmem_invalidate) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL(gmem_free_folio) +#endif #endif #undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +3 -1
@@ -1901,7 +1901,9 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); -void (*gmem_invalidate)(kvm_pfn_t start,kvm_pfn_t end); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void (*gmem_free_folio)(struct folio *folio); +#endif int (*gmem_max_mapping_level)(struct kvm *kvm,kvm_pfn_t pfn,bool is_private); };
arch/x86/kvm/svm/sev.c +3 -1
@@ -5136,8 +5136,10 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)return 0 ; } -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void sev_gmem_free_folio (struct folio *folio) { +kvm_pfn_t start =page_to_pfn (folio_page (folio,0 )); +kvm_pfn_t end = start + (1ul <<folio_order (folio));kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))
arch/x86/kvm/svm/svm.c +3 -1
@@ -5460,9 +5460,11 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, .alloc_apic_backing_page = svm_alloc_apic_backing_page, +#ifdef CONFIG_KVM_AMD_SEV .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, + .gmem_free_folio = sev_gmem_free_folio, .gmem_max_mapping_level = sev_gmem_max_mapping_level, +#endif };/*
arch/x86/kvm/svm/svm.h +1 -2
@@ -1009,7 +1009,7 @@ int sev_dev_get_attr(u32 group, u64 attr, u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code);int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); -void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +void sev_gmem_free_folio (struct folio *folio);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu);void sev_free_decrypted_vmsa (struct kvm_vcpu *vcpu,struct vmcb_save_area *vmsa);
@@ -1039,7 +1039,6 @@ static inline int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, in {return 0 ; } -static inline void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) {}static inline int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private) {return 0 ;
arch/x86/kvm/x86.c +2 -2
@@ -10592,9 +10592,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void kvm_arch_gmem_free_folio (struct folio *folio) { -kvm_x86_call (gmem_invalidate)(start, end); +kvm_x86_call (gmem_free_folio)(folio); }#endif #endif
include/linux/kvm_host.h +1 -1
@@ -2607,7 +2607,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end); +void kvm_arch_gmem_free_folio (struct folio *folio);#endif #ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY
virt/kvm/guest_memfd.c +1 -12
@@ -523,23 +523,12 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folreturn MF_DELAYED ; } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -static void kvm_gmem_free_folio (struct folio *folio) -{ -struct page *page =folio_page (folio,0 ); -kvm_pfn_t pfn =page_to_pfn (page); -int order =folio_order (folio); - -kvm_arch_gmem_invalidate (pfn, pfn + (1ul << order)); -} -#endif -static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE - .free_folio = kvm_gmem_free_folio, + .free_folio = kvm_arch_gmem_free_folio, #endif };
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Re: [PATCH v2 4/9] KVM: Rework .gmem_invalidate() into .gmem_free_folio()
arch/x86/kvm/svm/sev.c int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) · line 5145
Quoted 12 lines · Sean Christopherson
[...snip...]
arch/x86/kvm/svm/sev.c +3
-1-void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void sev_gmem_free_folio (struct folio *folio) { +kvm_pfn_t start =page_to_pfn (folio_page (folio,0 )); +kvm_pfn_t end = start + (1ul <<folio_order (folio));kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))
I thought we intended to draw the line such that the platforms don't reference folios, and so this function should be parametrized by pfn.
I think we should still stick with
.free_folio = kvm_gmem_free_folio
and kvm_gmem_free_folio() translates the folio to pfns and calls the arch function, named something like .gmem_LIFECYCLE_ACTION_pfn_range.
Now for LIFECYCLE_ACTION, one way to think of it is that this should represent the point in the lifecycle of guest_memfd memory where the memory is removed from guest's private use, so perhaps "host_reclaim"?
Then kvm_gmem_free_folio() becomes:
kvm_gmem_free_folio() {
pfn_start, pfn_end = translate folio to pfn range;
kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end);
}And in conversions
if (!to_private) {
pfn_start, pfn_end = translate guest_memfd offset range to pfns;
kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end);
}(and now it is right for the !to_private check to remain in guest_memfd since we're explicitly using that to guard a *host* reclaim function.
Another way to think of LIFECYCLE_ACTION is to have the hook represent the point of time where the memory's attribute is being set. Perhaps "set_attributes"?
Then kvm_gmem_free_folio() becomes:
kvm_gmem_free_folio() {
pfn_start, pfn_end = translate folio to pfn range;
kvm_x86_call(gmem_set_attributes_pfn_range)(pfn_start, pfn_end, SHARED);
}because when freeing a folio we want to set the attributes to SHARED.
And in conversions
if (kvm_x86_call(gmem_should_set_attributes_pfn_range)(SHARED)) {
pfn_start, pfn_end = translate guest_memfd offset range to pfns;
kvm_x86_call(gmem_set_attributes_pfn_range)(pfn_start, pfn_end, SHARED);
}This is biased towards conversions (related proposal at [1]).
[1] https://lore.kernel.org/all/CAEvNRgGX3GkazCWM=6y9YLgn=YemXuG==Oo+L58cac1Fd86_TQ@mail.gmail.com/
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v2 4/9] KVM: Rework .gmem_invalidate() into .gmem_free_folio()
Quoted 20 lines · Ackerley Tng
arch/x86/kvm/svm/sev.c int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) · line 5145
Quoted 12 lines · Sean Christopherson
[...snip...]
arch/x86/kvm/svm/sev.c +3
-1-void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void sev_gmem_free_folio (struct folio *folio) { +kvm_pfn_t start =page_to_pfn (folio_page (folio,0 )); +kvm_pfn_t end = start + (1ul <<folio_order (folio));kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))I thought we intended to draw the line such that the platforms don't reference folios, and so this function should be parametrized by pfn.
I think we should still stick with
Quoted 8 lines
.free_folio =
kvm_gmem_free_folioand
kvm_gmem_free_folio()translates the folio to pfns and calls the arch function, named something like .gmem_LIFECYCLE_ACTION_pfn_range.Now for
LIFECYCLE_ACTION, one way to think of it is that this should represent the point in the lifecycle ofguest_memfdmemory where the memory is removed from guest's private use, so perhaps "host_reclaim"?
kvm_arch_gmem_reclaim_memory()? I don't want to include "host", because the "reclaim" may or may not be host initiated. I don't want to use "pfn_range" because it's too close to "gfn_range".
Quoted 22 lines
Then
kvm_gmem_free_folio()becomes:kvm_gmem_free_folio() { pfn_start, pfn_end = translate folio to pfn range; kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end); }And in conversions
if (!to_private) { pfn_start, pfn_end = translate guest_memfd offset range to pfns; kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end); }(and now it is right for the !to_private check to remain in
guest_memfdsince we're explicitly using that to guard a *host* reclaim function.Another way to think of
LIFECYCLE_ACTIONis to have the hook represent the point of time where the memory's attribute is being set. Perhaps "set_attributes"?
Quoted 5 lines
Then
kvm_gmem_free_folio()becomes:kvm_gmem_free_folio() { pfn_start, pfn_end = translate folio to pfn range; kvm_x86_call(gmem_set_attributes_pfn_range)(pfn_start, pfn_end, SHARED);
No, because this isn't about setting memory PRIVATE vs. SHARED, it's about freeing memory back to the host. Which is why I suggested the comically verbose CONFIG_KVM_ARCH_GMEM_FREE_ON_SHARED_CONVERSION: SEV-SNP effectively frees memory when converting to shared.
Quoted 3 lines
}because when freeing a folio we want to set the attributes to SHARED.
No, because the RMP isn't being moved to a "shared" state per se, rather the page is being unassigned. And from KVM's perspective, freeing the folio can happen even if the memory isn't first converted to SHARED (in KVM's memory attributes).
Quoted 14 lines
And in conversions
if (kvm_x86_call(gmem_should_set_attributes_pfn_range)(SHARED)) { pfn_start, pfn_end = translate guest_memfd offset range to pfns; kvm_x86_call(gmem_set_attributes_pfn_range)(pfn_start, pfn_end, SHARED); }This is biased towards conversions (related proposal at [1]).
[1] https://lore.kernel.org/all/CAEvNRgGX3GkazCWM=6y9YLgn=YemXuG==Oo+L58cac1Fd86_TQ@mail.gmail.com/
Quoted 3 lines
[...snip...]
Re: [PATCH v2 4/9] KVM: Rework .gmem_invalidate() into .gmem_free_folio()
Quoted 26 lines · Sean Christopherson
Quoted 25 lines · Ackerley Tng
arch/x86/kvm/svm/sev.c int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) · line 5145
Quoted 12 lines · Sean Christopherson
[...snip...]
arch/x86/kvm/svm/sev.c +3
-1-void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void sev_gmem_free_folio (struct folio *folio) { +kvm_pfn_t start =page_to_pfn (folio_page (folio,0 )); +kvm_pfn_t end = start + (1ul <<folio_order (folio));kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))I thought we intended to draw the line such that the platforms don't reference folios, and so this function should be parametrized by pfn.
I think we should still stick with
.free_folio =
kvm_gmem_free_folioand
kvm_gmem_free_folio()translates the folio to pfns and calls the arch function, named something like .gmem_LIFECYCLE_ACTION_pfn_range.
Quoted 8 lines
Quoted 3 lines
Now for
LIFECYCLE_ACTION, one way to think of it is that this should represent the point in the lifecycle ofguest_memfdmemory where the memory is removed from guest's private use, so perhaps "host_reclaim"?
kvm_arch_gmem_reclaim_memory()? I don't want to include "host", because the "reclaim" may or may not be host initiated. I don't want to use "pfn_range" because it's too close to "gfn_range".
kvm_arch_gmem_reclaim_memory() is good. It'll take PFNs pfn_start and pfn_end? If yes, I'll wait for your next revision before I make a v9 for guest_memfd in-place conversions.
I think Fuad and Suzuki should be good with this too wrt conversions, based on their responses to [1].
Quoted 27 lines
Quoted 27 lines
Then
kvm_gmem_free_folio()becomes:kvm_gmem_free_folio() { pfn_start, pfn_end = translate folio to pfn range; kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end); }And in conversions
if (!to_private) { pfn_start, pfn_end = translate guest_memfd offset range to pfns; kvm_x86_call(gmem_host_reclaim_pfn_range)(pfn_start, pfn_end); }(and now it is right for the !to_private check to remain in
guest_memfdsince we're explicitly using that to guard a *host* reclaim function.[...snip...]
[1] https://lore.kernel.org/all/CAEvNRgGX3GkazCWM=6y9YLgn=YemXuG==Oo+L58cac1Fd86_TQ@mail.gmail.com/
Quoted 3 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v2 4/9] KVM: Rework .gmem_invalidate() into .gmem_free_folio()
Quoted 32 lines · Ackerley Tng
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Ackerley Tng
arch/x86/kvm/svm/sev.c int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) · line 5145
Quoted 12 lines · Sean Christopherson
[...snip...]
arch/x86/kvm/svm/sev.c +3
-1-void sev_gmem_invalidate (kvm_pfn_t start,kvm_pfn_t end) +void sev_gmem_free_folio (struct folio *folio) { +kvm_pfn_t start =page_to_pfn (folio_page (folio,0 )); +kvm_pfn_t end = start + (1ul <<folio_order (folio));kvm_pfn_t pfn;if (!cc_platform_has (CC_ATTR_HOST_SEV_SNP ))I thought we intended to draw the line such that the platforms don't reference folios, and so this function should be parametrized by pfn.
I think we should still stick with
.free_folio =
kvm_gmem_free_folioand
kvm_gmem_free_folio()translates the folio to pfns and calls the arch function, named something like .gmem_LIFECYCLE_ACTION_pfn_range.Now for
LIFECYCLE_ACTION, one way to think of it is that this should represent the point in the lifecycle ofguest_memfdmemory where the memory is removed from guest's private use, so perhaps "host_reclaim"?
Quoted 7 lines
Quoted 4 lines
kvm_arch_gmem_reclaim_memory()? I don't want to include "host", because the "reclaim" may or may not be host initiated. I don't want to use "pfn_range" because it's too close to "gfn_range".
kvm_arch_gmem_reclaim_memory()is good. It'll take PFNspfn_startandpfn_end?
Yep, I'll just do a pure rename.
Sean Christopherson
maintainerPATCH v2 5/9 KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into kvm_arch_flush_shadow_memslot()
In reply to Sean ChristophersonFold kvm_mmu_zap_memslot() into its sole caller so that its GFN range structure can be used to trigger guest_memfd invalidations regardless of whether KVM will do a partial or full zap of the MMU.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -20
@@ -7560,8 +7560,14 @@ static void kvm_mmu_zap_memslot_pages_and_flush(struct kvm *kvm,kvm_mmu_remote_flush_or_zap (kvm, &invalid_list, flush); } -static void kvm_mmu_zap_memslot (struct kvm *kvm, -struct kvm_memory_slot *slot) +static inline bool kvm_memslot_flush_zap_all (struct kvm *kvm) +{ +return kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && +kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL ); +} + +void kvm_arch_flush_shadow_memslot (struct kvm *kvm, +struct kvm_memory_slot *slot) {struct kvm_gfn_range range = { .slot = slot,
@@ -7572,25 +7578,14 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, };bool flush; -write_lock (&kvm->mmu_lock); - flush =kvm_unmap_gfn_range (kvm, &range); -kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); -write_unlock (&kvm->mmu_lock); -} - -static inline bool kvm_memslot_flush_zap_all (struct kvm *kvm) -{ -return kvm->arch.vm_type ==KVM_X86_DEFAULT_VM && -kvm_check_has_quirk (kvm,KVM_X86_QUIRK_SLOT_ZAP_ALL ); -} - -void kvm_arch_flush_shadow_memslot (struct kvm *kvm, -struct kvm_memory_slot *slot) -{ -if (kvm_memslot_flush_zap_all (kvm)) +if (kvm_memslot_flush_zap_all (kvm)) {kvm_mmu_zap_all_fast (kvm); -else -kvm_mmu_zap_memslot (kvm, slot); + }else { +write_lock (&kvm->mmu_lock); + flush =kvm_unmap_gfn_range (kvm, &range); +kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); +write_unlock (&kvm->mmu_lock); + } }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 6/9 KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves
In reply to Sean ChristophersonSplit kvm_mmu_zap_all_fast() into a "front half" and a "back half", where the front half is everything that runs with mmu_lock held for write, and the back half is the code that runs outside of mmu_lock.
Manually invoke the two halves when zapping in response to a memslot being DELETE or MOVED, and share the mmu_lock critical section between the "fast zap all" and "zap only the memslot" paths. This will allow putting more code inside the critical section without having to take mmu_lock twice in quick succession.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 48 +++++++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/mmu.c +33 -15
@@ -6921,20 +6921,11 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)kvm_mmu_commit_zap_page (kvm, &invalid_list); } -/* -* Fast invalidate all shadow pages and use lock-break technique -* to zap obsolete pages. -* -* It's required when memslot is being deleted or VM is being -* destroyed, in these cases, we should ensure that KVM MMU does -* not use any resource of the being-deleted slot or all slots -* after calling the function. -*/ -static void kvm_mmu_zap_all_fast (struct kvm *kvm) +static void __kvm_mmu_zap_all_fast_front_half (struct kvm *kvm) {lockdep_assert_held (&kvm->slots_lock); +lockdep_assert_held_write (&kvm->mmu_lock); -write_lock (&kvm->mmu_lock);trace_kvm_mmu_zap_all_fast (kvm);/*
@@ -6971,8 +6962,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_make_all_cpus_request (kvm,KVM_REQ_MMU_FREE_OBSOLETE_ROOTS );kvm_zap_obsolete_pages (kvm); +} -write_unlock (&kvm->mmu_lock); +static void __kvm_mmu_zap_all_fast_back_half (struct kvm *kvm) +{ +lockdep_assert_held (&kvm->slots_lock); +lockdep_assert_not_held (&kvm->mmu_lock);/* * Zap the invalidated TDP MMU roots, all SPTEs must be dropped before
@@ -6986,6 +6981,24 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_tdp_mmu_zap_invalidated_roots (kvm, true); } +/* +* Fast invalidate all shadow pages and use lock-break technique +* to zap obsolete pages. +* +* It's required when memslot is being deleted or VM is being +* destroyed, in these cases, we should ensure that KVM MMU does +* not use any resource of the being-deleted slot or all slots +* after calling the function. +*/ +static void kvm_mmu_zap_all_fast (struct kvm *kvm) +{ +write_lock (&kvm->mmu_lock); +__kvm_mmu_zap_all_fast_front_half (kvm); +write_unlock (&kvm->mmu_lock); + +__kvm_mmu_zap_all_fast_back_half (kvm); +} +int kvm_mmu_init_vm (struct kvm *kvm) {int r, i;
@@ -7578,14 +7591,19 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, };bool flush; +write_lock (&kvm->mmu_lock); +if (kvm_memslot_flush_zap_all (kvm)) { -kvm_mmu_zap_all_fast (kvm); +__kvm_mmu_zap_all_fast_front_half (kvm); }else { -write_lock (&kvm->mmu_lock); flush =kvm_unmap_gfn_range (kvm, &range);kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); -write_unlock (&kvm->mmu_lock); } + +write_unlock (&kvm->mmu_lock); + +if (kvm_memslot_flush_zap_all (kvm)) +__kvm_mmu_zap_all_fast_back_half (kvm); }void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen)
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
[email protected]
AI reviewRe: [PATCH v2 6/9] KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Evaluating `kvm_memslot_flush_zap_all(kvm)` twice in `kvm_arch_flush_shadow_memslot()` creates…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Evaluating kvm_memslot_flush_zap_all(kvm) twice in kvm_arch_flush_shadow_memslot() creates a TOCTOU race window that allows concurrent quirk modification to skip the back-half zap, leading to a Use-After-Free of host memory. --
commit daadba9269f9868eb59f2fe4d2fd49715b55bc67
Author: Sean Christopherson <[email protected]>
KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves
This commit splits kvm_mmu_zap_all_fast() into a front half (running with mmu_lock held for write) and a back half (running outside of mmu_lock). This allows sharing the mmu_lock critical section when zapping in response to a memslot being deleted or moved.
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +0
-0
[ ... ]
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c +3
-2@@ -7578,14 +7591,19 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, };bool flush; +write_lock (&kvm->mmu_lock); +if (kvm_memslot_flush_zap_all (kvm)) { -kvm_mmu_zap_all_fast (kvm); +__kvm_mmu_zap_all_fast_front_half (kvm); }else { -write_lock (&kvm->mmu_lock);
arch/x86/kvm/mmu/mmu.c void kvm_arch_flush_shadow_memslot(struct kvm *kvm, · line 7607
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +5
-1flush =kvm_unmap_gfn_range (kvm, &range);kvm_mmu_zap_memslot_pages_and_flush (kvm, slot, flush); -write_unlock (&kvm->mmu_lock); } + +write_unlock (&kvm->mmu_lock); + +if (kvm_memslot_flush_zap_all (kvm)) +__kvm_mmu_zap_all_fast_back_half (kvm); }
[Severity: Critical]
Can kvm_memslot_flush_zap_all() evaluate differently on the second check in kvm_arch_flush_shadow_memslot()?
Since kvm_memslot_flush_zap_all() dynamically checks disabled_quirks, if a concurrent thread disables KVM_X86_QUIRK_SLOT_ZAP_ALL after the front half executes but before the back half check:
Thread 1 in kvm_arch_flush_shadow_memslot():
if (kvm_memslot_flush_zap_all(kvm))
__kvm_mmu_zap_all_fast_front_half(kvm);Thread 2:
KVM_ENABLE_CAP with KVM_CAP_DISABLE_QUIRKS
(disables KVM_X86_QUIRK_SLOT_ZAP_ALL)
Thread 1 resumes:
if (kvm_memslot_flush_zap_all(kvm))
// Evaluates to false, skips back half
__kvm_mmu_zap_all_fast_back_half(kvm);Could this skip the back half zap and leave invalidated TDP MMU roots unzapped? If so, this could lead to a use-after-free because KVM's MMU notifiers will not find the mapping to zap it when the host frees the memory.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Sean Christopherson
maintainerPATCH v2 7/9 KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
In reply to Sean ChristophersonWire up a gmem_invalid_range() call for SNP VMs, and use it to force vCPUs to reload/recheck their guest-provided VMSA if the backing guest_memfd page is being invalidated, e.g. is being PUNCH_HOLE'd. Use the same core logic to handle invalidations as VMX does for the APIC-access page, as the two concepts are nearly identical: shove the physical address of a page into the vCPU's control structure:
1. Snapshot the invalidation sequence counter
2. Grab the pfn (from guest_memfd in this case)
3. Acquire mmu_lock for read
4. Re-request reload if retry is needed, otherwise commit the change.
Note, the re-request action in #4 is necessary as KVM's retry logic is fuzzy, i.e. can get false positives. If the guest_memfd page has been dropped, at some point a subsequent reload will fail to get a PFN from guest_memfd, and KVM will fail KVM_RUN. If the retry was due to a false positive, KVM will retry until there are no relevant MMU notifier events (and will retry in the "outer" loop, i.e. will drop locks and resched as needed).
Note #2! Take care to invalidate the VMSA when a relevant memslot is DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated on memslot bindings (KVM doesn't know what GFN range(s) to invalidate without a binding). And more importantly, the VMSA mapping requires a memslot, i.e. must be invalidated if its memslots disappears, regardless of the state of the underlying guest_memfd inode.
Failure to invalidate the vCPU's control.vmsa_pa (which is checked by pre_sev_run()) can prevent KVM from properly freeing the page as firmware will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU is actively running, i.e. if VMSA page is in-use. That in turn leads to an RMP #PF on the next use, as the page will still be assigned to the SNP VM.
SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O Tainted: [U]=USER, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 Call Trace: <TASK> dump_stack_lvl+0x54/0x70 rmpupdate+0x12c/0x140 rmp_make_shared+0x3b/0x60 sev_gmem_invalidate+0xe0/0x170 [kvm_amd] delete_from_page_cache_batch+0x1d8/0x220 truncate_inode_pages_range+0x120/0x3d0 kvm_gmem_fallocate+0x19a/0x270 [kvm] vfs_fallocate+0x1bc/0x1f0 __x64_sys_fallocate+0x48/0x70 do_syscall_64+0x10a/0x480 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x496c7e </TASK> ------------[ cut here ]------------ SEV: Failed to update RMP entry for PFN 0x78d198 error -14 WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345 CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O Tainted: [U]=USER, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd] Call Trace: <TASK> delete_from_page_cache_batch+0x1d8/0x220 truncate_inode_pages_range+0x120/0x3d0 kvm_gmem_fallocate+0x19a/0x270 [kvm] vfs_fallocate+0x1bc/0x1f0 __x64_sys_fallocate+0x48/0x70 do_syscall_64+0x10a/0x480 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x496c7e </TASK> irq event stamp: 20689 hardirqs last enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60 hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60 softirqs last enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140 softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140 ---[ end trace 0000000000000000 ]--- BUG: unable to handle page fault for address: ffff99a64d198000 #PF: supervisor write access in kernel mode #PF: error_code(0x80000003) - RMP violation PGD 13eb001067 P4D 13eb001067 PUD 78d1d1063 PMD 1184e0063 PTE 800000078d198163 SEV-SNP: PFN 0x78d198, RMP entry: [0x6030000000144001 - 0x000000000000000f] Oops: Oops: 0003 [#1] SMP CPU: 3 UID: 0 PID: 31407 Comm: highlanderd_hea Tainted: G U W O Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026 RIP: 0010:prep_new_page+0x67/0x220 Call Trace: <TASK> get_page_from_freelist+0x1c40/0x1c70 __alloc_frozen_pages_noprof+0xca/0x1f0 alloc_pages_mpol+0x10b/0x1b0 alloc_pages_noprof+0x81/0x90 pte_alloc_one+0x1b/0xd0 do_pte_missing+0xdf/0x1020 handle_mm_fault+0x7c7/0xb20 do_user_addr_fault+0x268/0x6b0 exc_page_fault+0x67/0xa0 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x4a6b1e </TASK> gsmi: Log Shutdown Reason 0x03 CR2: ffff99a64d198000 ---[ end trace 0000000000000000 ]--- RIP: 0010:prep_new_page+0x67/0x220
Drop the pseudo-TODO comment about needing to pin the page if guest_memfd every supports migration, as integrating with invalidations events means KVM will Just Work if/when page migration is ever supported (assuming SNP hardware supports migrating VMSA pages).
--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mmu/mmu.c | 5 +++ arch/x86/kvm/svm/sev.c | 62 +++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.c | 2 + arch/x86/kvm/svm/svm.h | 2 + arch/x86/kvm/x86.c | 6 +++ include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 4 ++ 9 files changed, 79 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL(reload_vmsa) KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked)
@@ -148,6 +149,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL(gmem_invalidate_range) KVM_X86_OP_OPTIONAL(gmem_free_folio)#endif #endif
arch/x86/include/asm/kvm_host.h +4 -0
@@ -122,6 +122,8 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_VMSA_PAGE_RELOAD \ + KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \ KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
@@ -1878,6 +1880,7 @@ struct kvm_x86_ops {int (*vm_copy_enc_context_from)(struct kvm *kvm,unsigned int source_fd);int (*vm_move_enc_context_from)(struct kvm *kvm,unsigned int source_fd);void (*guest_memory_reclaimed)(struct kvm *kvm); +void (*reload_vmsa)(struct kvm_vcpu *vcpu);int (*get_feature_msr)(u32 msr,u64 *data);
@@ -1902,6 +1905,7 @@ struct kvm_x86_ops {void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order);#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range);void (*gmem_free_folio)(struct folio *folio);#endif int (*gmem_max_mapping_level)(struct kvm *kvm,kvm_pfn_t pfn,bool is_private);
arch/x86/kvm/mmu/mmu.c +5 -0
@@ -7593,6 +7593,11 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,write_lock (&kvm->mmu_lock); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +if (slot->gmem.file) +kvm_arch_gmem_invalidate_range (kvm, &range); +#endif +if (kvm_memslot_flush_zap_all (kvm)) {__kvm_mmu_zap_all_fast_front_half (kvm); }else {
arch/x86/kvm/svm/sev.c +53 -9
104 lines not on this page · raw message
arch/x86/kvm/svm/svm.c +2 -0
@@ -5445,6 +5445,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .mem_enc_register_region = sev_mem_enc_register_region, .mem_enc_unregister_region = sev_mem_enc_unregister_region, .guest_memory_reclaimed = sev_guest_memory_reclaimed, + .reload_vmsa = sev_snp_reload_vmsa, .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from,
@@ -5462,6 +5463,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {#ifdef CONFIG_KVM_AMD_SEV .gmem_prepare = sev_gmem_prepare, + .gmem_invalidate_range = sev_gmem_invalidate_range, .gmem_free_folio = sev_gmem_free_folio, .gmem_max_mapping_level = sev_gmem_max_mapping_level,#endif
arch/x86/kvm/svm/svm.h +2 -0
@@ -996,6 +996,7 @@ static inline struct page *snp_safe_alloc_page(void) {return snp_safe_alloc_page_node (numa_node_id (),GFP_KERNEL_ACCOUNT ); } +void sev_snp_reload_vmsa (struct kvm_vcpu *vcpu);int sev_vcpu_create (struct kvm_vcpu *vcpu);void sev_free_vcpu (struct kvm_vcpu *vcpu);
@@ -1009,6 +1010,7 @@ int sev_dev_get_attr(u32 group, u64 attr, u64 *val);extern unsigned int max_sev_asid;void sev_handle_rmp_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code);int sev_gmem_prepare (struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);void sev_gmem_free_folio (struct folio *folio);int sev_gmem_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn,bool is_private);struct vmcb_save_area *sev_decrypt_vmsa (struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +6 -0
@@ -8167,6 +8167,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } +if (kvm_check_request (KVM_REQ_VMSA_PAGE_RELOAD , vcpu)) +kvm_x86_call (reload_vmsa)(vcpu); } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
@@ -10592,6 +10594,10 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_x86_call (gmem_invalidate_range)(kvm, range); +}void kvm_arch_gmem_free_folio (struct folio *folio) {kvm_x86_call (gmem_free_folio)(folio);
include/linux/kvm_host.h +1 -0
@@ -2607,6 +2607,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void kvm_arch_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range);void kvm_arch_gmem_free_folio (struct folio *folio);#endif
virt/kvm/guest_memfd.c +4 -0
@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif }if (flush)
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
[email protected]
AI reviewRe: [PATCH v2 7/9] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Data Race on lockless access to `snp_guest_vmsa_gpa` without READ_ONCE/WRITE_ONCE. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Data Race on lockless access to snp_guest_vmsa_gpa without READ_ONCE/WRITE_ONCE. --
commit 0384ec40691742221905a1f869e903471b8acac7
Author: Sean Christopherson <[email protected]>
KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
This commit wires up a gmem_invalid_range() call for SNP VMs. It forces vCPUs to reload or recheck their guest-provided VMSA if the backing guest_memfd page is being invalidated.
Quoted 4 lines
arch/x86/kvm/svm/sev.c +0
-0
[ ... ]
Quoted 14 lines
arch/x86/kvm/svm/sev.c +10
-0@@ -5135,6 +5154,31 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)return 0 ; } +void sev_gmem_invalidate_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +struct kvm_vcpu *vcpu; +unsigned long i; + +lockdep_assert_held_write (&kvm->mmu_lock); + +/* +* An unstable result for "is SNP" is a-ok here, thanks to mmu_lock. +* The vCPU's VMSA GPA is invalidated before the vCPU is made visible
arch/x86/kvm/svm/sev.c int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) · line 5176
Quoted 10 lines
arch/x86/kvm/svm/sev.c +10
-0+* to other tasks, and can only become valid while holding mmu_lock, +* after the VM is fully committed to being an SNP VM. +*/ +if (!____sev_snp_guest (kvm)) +return ; + +kvm_for_each_vcpu (i, vcpu, kvm) { +gpa_t gpa =to_svm (vcpu)->sev_es.snp_guest_vmsa_gpa; + + if (VALID_PAGE(gpa) &&
[Severity: Low]
Can this lead to a data race or KCSAN splat?
Here snp_guest_vmsa_gpa is read locklessly relative to snp_vmsa_mutex. It is read under mmu_lock, but in __sev_snp_reload_vmsa(), snp_guest_vmsa_gpa is cleared (set to INVALID_PAGE) while holding snp_vmsa_mutex without mmu_lock.
Since they don't share a common lock for this variable's lifecycle, should this read and the corresponding write in __sev_snp_reload_vmsa() use READ_ONCE() and WRITE_ONCE() to prevent compiler refetching or optimization artifacts?
Quoted 5 lines
arch/x86/kvm/svm/sev.c +5
-0+ gpa_to_gfn(gpa) >= range->start && + gpa_to_gfn(gpa) < range->end) +kvm_make_request_and_kick (KVM_REQ_VMSA_PAGE_RELOAD , vcpu); + } +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Re: [PATCH v2 7/9] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
In reply to Sean Christophersonarch/x86/include/asm/kvm-x86-ops.h KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) · line 137
Quoted 12 lines · Sean Christopherson
[...snip...]
arch/x86/include/asm/kvm-x86-ops.h +1
-0@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL(reload_vmsa)
Regarding naming, this seems to be the most platform-specific KVM_X86_OP there is (VMSA is a SEV-ES/SNP only thing), though I can't think of a way to make this seem more general, to make it stick out less among the x86 ops. Maybe it's not necessary to make it blend in anyway.
arch/x86/include/asm/kvm-x86-ops.h KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) · line 152
Quoted 8 lines
arch/x86/include/asm/kvm-x86-ops.h +1
-0KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked)@@ -148,6 +149,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
Clarification: .gmem_invalidate_range() should be thought of as "this is the hook called when some guest_memfd memory is invalidated" (as in, this is a point in the lifecycle of guest_memfd memory) and not "this hook invalidates guest memory in this range", right?
Quoted 11 lines
arch/x86/include/asm/kvm-x86-ops.h +0
-0KVM_X86_OP_OPTIONAL(gmem_free_folio)#endif #endif arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8171
Quoted 5 lines
arch/x86/kvm/x86.c +2
-0@@ -8167,6 +8167,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } +if (kvm_check_request (KVM_REQ_VMSA_PAGE_RELOAD , vcpu)) +kvm_x86_call (reload_vmsa)(vcpu);
Should this be wrapped with an #ifdef to skip the check entirely for non-SNP platforms?
Quoted 14 lines
arch/x86/kvm/x86.c +0
-0} if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||@@ -10592,6 +10594,10 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord#endif virt/kvm/guest_memfd.c +0
-0
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 191
Quoted 7 lines
virt/kvm/guest_memfd.c +4
-0@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif
In general, how do you think of using stubs vs @ifdefs?
Would it be better to define a stub for kvm_arch_gmem_invalidate_range() that does nothing #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE?
Quoted 5 lines
virt/kvm/guest_memfd.c +0
-1}if (flush) --2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerRe: [PATCH v2 7/9] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
Quoted 5 lines · Ackerley Tng
Quoted 3 lines · Sean Christopherson
[...snip...]
Quoted 14 lines
arch/x86/include/asm/kvm-x86-ops.h KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) · line 137
Quoted 9 lines
arch/x86/include/asm/kvm-x86-ops.h +1
-0@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL(reload_vmsa)Regarding naming, this seems to be the most platform-specific
KVM_X86_OPthere is (VMSA is a SEV-ES/SNP only thing), though I can't think of a way to make this seem more general, to make it stick out less among the x86 ops. Maybe it's not necessary to make it blend in anyway.
Yeah, it sucks. I wanted to turn KVM_REQ_APIC_PAGE_RELOAD into a generic "reload vendor specific thing" request, but it's not feasible to do that without a pile of ugly conditionals (or potential false positives on AMD).
Quoted 5 lines
Quoted 5 lines
arch/x86/include/asm/kvm-x86-ops.h +0
-0KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked)@@ -148,6 +149,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)KVM_X86_OP_OPTIONAL_RET0 (gmem_prepare)
Quoted 8 lines
arch/x86/include/asm/kvm-x86-ops.h KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) · line 152
Quoted 3 lines
arch/x86/include/asm/kvm-x86-ops.h +1
-0KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL (gmem_invalidate_range)Clarification: .gmem_invalidate_range() should be thought of as "this is the hook called when some
guest_memfdmemory is invalidated" (as in, this is a point in the lifecycle ofguest_memfdmemory) and not "this hook invalidates guest memory in this range", right?
Yes, the former, though I would phrase it more along the lines of "something about this memory may be changing, drop all references".
Quoted 6 lines
Quoted 6 lines
arch/x86/include/asm/kvm-x86-ops.h +0
-0KVM_X86_OP_OPTIONAL(gmem_free_folio)#endif #endif
Quoted 13 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8171
Quoted 10 lines
arch/x86/kvm/x86.c +2
-0@@ -8167,6 +8167,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } +if (kvm_check_request (KVM_REQ_VMSA_PAGE_RELOAD , vcpu)) +kvm_x86_call (reload_vmsa)(vcpu);Should this be wrapped with an #ifdef to skip the check entirely for non-SNP platforms?
No, we generally avoid using #ifdefs as micro-optimizations, and this would be more like a pico-optimazation. It's a single uop that in practice will be easily predicted by hardware.
Quoted 16 lines
Quoted 16 lines
arch/x86/kvm/x86.c +0
-0} if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||@@ -10592,6 +10594,10 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord#endif virt/kvm/guest_memfd.c +0
-0@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, }
Quoted 7 lines
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 191
Quoted 5 lines
virt/kvm/guest_memfd.c +4
-0flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +kvm_arch_gmem_invalidate_range (kvm, &gfn_range); +#endif In general, how do you think of using stubs vs @ifdefs?
They both have their uses. Generally speaking, KVM (and x86 in general) tries to avoid #ifdefs as they tend to make the code harder to read. But there are cases where stubs simply don't make sense, e.g. all of the code guarded by CONFIG_KVM_XEN, CONFIG_KVM_IOAPIC, and CONFIG_KVM_HYPERV is so tightly coupled to its specific "thing" that using a stub makes very little sense. And providing stubs can be actively dangerous, in that it allows selecting a Kconfig without wiring up the plumbing, whereas using #ifdefs will result in build failures.
Quoted 2 lines
Would it be better to define a stub for
kvm_arch_gmem_invalidate_range()that does nothing #ifdefCONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE?
I don't have a super strong preference, but given that the whole point of the Kconfig is to opt-in to calling an arch hook, providing a stub falls into the "actively dangerous" category.
And I don't want to diverge from what we do for CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE, i.e. *if* we want to provide stubs instead of using #ifdefs, then I want to do that separately, in a concerted effort.
Sean Christopherson
maintainerPATCH v2 8/9 KVM: x86: Guard .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE=y
In reply to Sean ChristophersonWrap the .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE so that non-SEV code doesn't try to wire up a callback without doing the necessary enabling.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 +++- arch/x86/include/asm/kvm_host.h | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +3 -1
@@ -146,12 +146,14 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) -KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) +#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE KVM_X86_OP_OPTIONAL(gmem_invalidate_range) KVM_X86_OP_OPTIONAL(gmem_free_folio) #endif +KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) #endif #undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1903,7 +1903,9 @@ struct kvm_x86_ops {gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu,gva_t gva,unsigned int flags);void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE int (*gmem_prepare)(struct kvm *kvm,kvm_pfn_t pfn,gfn_t gfn,int max_order); +#endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE void (*gmem_invalidate_range)(struct kvm *kvm,struct kvm_gfn_range *range);void (*gmem_free_folio)(struct folio *folio);
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 9/9 KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid
In reply to Sean ChristophersonTrack the guest as having a guest-provided VMSA as soon as control.vmsa_pa is invalidated, instead of waiting to see if the guest-provided VMSA is usable, so that KVM doesn't switch back to the original VMSA instead of exiting to userspace (due to an invalid VMSA). By the time a vCPU tries to load a guest-provided VMSA, KVM has already communicated "success" for AP creation, i.e. KVM has committed to using the guest-provided VMSA.
--- arch/x86/kvm/svm/sev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
arch/x86/kvm/svm/sev.c +17 -17
@@ -4001,23 +4001,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)*/ vmcb_mark_all_dirty (svm->vmcb); -if (!VALID_PAGE (gpa)) -return ; - - slot =gfn_to_memslot (vcpu->kvm, gfn); -if (!slot) -return ; - - mmu_seq = kvm->mmu_invalidate_seq; -smp_rmb (); - -/* -* The new VMSA will be private memory guest memory, so retrieve the -* PFN from the gmem backend. -*/ -if (kvm_gmem_get_pfn (vcpu->kvm, slot, gfn, &pfn, &page,NULL )) -return ; -/* * From this point forward, the VMSA will always be a guest-mapped page * rather than the initial one allocated by KVM in svm->sev_es.vmsa. In
@@ -4029,6 +4012,23 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)*/ svm->sev_es.snp_has_guest_vmsa = true; +if (!VALID_PAGE (gpa)) +return ; + + slot =gfn_to_memslot (vcpu->kvm, gfn); +if (!slot) +return ; + + mmu_seq = kvm->mmu_invalidate_seq; +smp_rmb (); + +/* +* The new VMSA will be private memory guest memory, so retrieve the +* PFN from the gmem backend. +*/ +if (kvm_gmem_get_pfn (vcpu->kvm, slot, gfn, &pfn, &page,NULL )) +return ; +read_lock (&kvm->mmu_lock);/* * Save the guest-provided GPA. If retry is needed, then KVM will try
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog