KVM: x86/SVM: Fixes for AMD ERAPS virtualization
v2 cover KVM: x86/SVM: Fixes for AMD ERAPS virtualization
This series fixes three issues in KVM's virtualization of Enhanced Return Address Predictor Security (ERAPS):
Patch 1 adds ALLOW_LARGER_RAP configuration to svm_vcpu_after_set_cpuid() so that the BSP will be properly configured. The ALLOW_LARGER_RAP configuration in init_vmcb() is retained to restore the bit after the VMCB is cleared on the VCPU triple fault path.
Patch 2 moves ERAPS dirty tracking into svm_flush_tlb_asid(). AMD ERAPS architecturally requires clearing the Return Address Predictor (RAP/RSB) on all implicit TLB invalidations. Centralizing dirty tracking in svm_flush_tlb_asid() ensures that local/current ASID flushes (e.g. same-value CR3 reloads under shadow paging, CR4.PAE/SMEP toggles, single-context INVPCID) correctly mark ERAPS dirty and clear the RAP on the next VMRUN.
Patch 3 adds a KVM_REQ_TLB_FLUSH_GUEST request to kvm_mtrr_set_msr() when MTRR MSRs are written via WRMSR. Per AMD APM and Intel SDM, MTRR writes are implicit TLB invalidations. Requesting a guest TLB flush restores x86 architectural compliance and ensures ERAPS RAP clearing on MTRR writes.
v1 -> v2:
* Restored the ALLOW_LARGER_RAP configuration in init_vmcb() [Sashiko]
v1: https://lore.kernel.org/kvm/[email protected]/
Jim Mattson (3):
KVM: SVM: Configure ALLOW_LARGER_RAP in svm_vcpu_after_set_cpuid()
KVM: SVM: Dirty ERAPS register on all ASID TLB flushes
KVM: x86: Flush guest TLB on MTRR MSR writes
arch/x86/kvm/mtrr.c | 1 + arch/x86/kvm/svm/svm.c | 16 ++++++++-------- 2 files changed, 9 insertions(+), 8 deletions(-)
base-commit: 58717b2a1365d06c8c64b72aa948541b53fe31eb
Signature · 1 line
2.55.0.229.g6434b31f56-goog
PATCH v2 1/3 KVM: SVM: Configure ALLOW_LARGER_RAP in svm_vcpu_after_set_cpuid()
On initial vCPU creation, init_vmcb() is called before userspace configures guest CPUID via KVM_SET_CPUID2. Consequently, ERAP_CONTROL_ALLOW_LARGER_RAP is never set during initial vCPU creation. If an AP later receives an INIT signal after CPUID has been set, init_vmcb() would set the bit for that AP, but the BSP would be left with ALLOW_LARGER_RAP clear.
Configure ALLOW_LARGER_RAP in svm_vcpu_after_set_cpuid() to handle the BSP.
Keep the code in init_vmcb() to restore ALLOW_LARGER_RAP after the VMCB has been cleared on the VCPU triple-fault reset path.
--- arch/x86/kvm/svm/svm.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/kvm/svm/svm.c +5 -0
@@ -4736,6 +4736,11 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (guest_cpuid_is_intel_compatible (vcpu))guest_cpu_cap_clear (vcpu,X86_FEATURE_V_VMSAVE_VMLOAD ); +if (guest_cpu_cap_has (vcpu,X86_FEATURE_ERAPS )) + svm->vmcb01.ptr->control.erap_ctl |=ERAP_CONTROL_ALLOW_LARGER_RAP ; +else + svm->vmcb01.ptr->control.erap_ctl &= ~ERAP_CONTROL_ALLOW_LARGER_RAP ; +if (is_sev_guest (vcpu))sev_vcpu_after_set_cpuid (svm); }
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Sean Christopherson
maintainerRe: [PATCH v2 1/3] KVM: SVM: Configure ALLOW_LARGER_RAP in svm_vcpu_after_set_cpuid()
Quoted 23 lines · Jim Mattson
On initial vCPU creation,
init_vmcb()is called before userspace configures guest CPUID viaKVM_SET_CPUID2. Consequently,ERAP_CONTROL_ALLOW_LARGER_RAPis never set during initial vCPU creation. If an AP later receives an INIT signal after CPUID has been set,init_vmcb()would set the bit for that AP, but the BSP would be left withALLOW_LARGER_RAPclear.Configure
ALLOW_LARGER_RAPinsvm_vcpu_after_set_cpuid()to handle the BSP.Keep the code in
init_vmcb()to restoreALLOW_LARGER_RAPafter the VMCB has been cleared on the VCPU triple-fault reset path.Fixes db5e82496492 ("KVM: SVM: Virtualize and advertise support for ERAPS")Assisted-by Gemini:Gemini-NextSigned-off-by Jim Mattson <[email protected]>--- arch/x86/kvm/svm/svm.c | 5 +++++ 1 file changed, 5 insertions(+)arch/x86/kvm/svm/svm.c +0
-0
arch/x86/kvm/svm/svm.c static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 4742
Quoted 7 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -4736,6 +4736,11 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (guest_cpuid_is_intel_compatible (vcpu))guest_cpu_cap_clear (vcpu,X86_FEATURE_V_VMSAVE_VMLOAD ); +if (guest_cpu_cap_has (vcpu,X86_FEATURE_ERAPS )) + svm->vmcb01.ptr->control.erap_ctl |=ERAP_CONTROL_ALLOW_LARGER_RAP ; +else + svm->vmcb01.ptr->control.erap_ctl &= ~ERAP_CONTROL_ALLOW_LARGER_RAP ;
If we handle ERAPS here, can't we drop the code in init_vmcb()? In the context of PML enablement, AMD is updating the APM to clarify that the control area remains valid after shutdown, i.e. KVM doesn't need to re-initialize erap_ctl when synthesizing INIT from shutdown_interception().
"After an intercepted shutdown, the VMCB control area is valid (with the
exception of offsets 60h, 61h, and 68h) and the VMCB state save area is undefined."
Quoted 7 lines
arch/x86/kvm/svm/svm.c +1
-0+if (is_sev_guest (vcpu))sev_vcpu_after_set_cpuid (svm); }Signature · 1 line
2.55.0.229.g6434b31f56-goog
Sean Christopherson
maintainerRe: [PATCH v2 1/3] KVM: SVM: Configure ALLOW_LARGER_RAP in svm_vcpu_after_set_cpuid()
Quoted 37 lines · Sean Christopherson
Quoted 23 lines · Jim Mattson
On initial vCPU creation,
init_vmcb()is called before userspace configures guest CPUID viaKVM_SET_CPUID2. Consequently,ERAP_CONTROL_ALLOW_LARGER_RAPis never set during initial vCPU creation. If an AP later receives an INIT signal after CPUID has been set,init_vmcb()would set the bit for that AP, but the BSP would be left withALLOW_LARGER_RAPclear.Configure
ALLOW_LARGER_RAPinsvm_vcpu_after_set_cpuid()to handle the BSP.Keep the code in
init_vmcb()to restoreALLOW_LARGER_RAPafter the VMCB has been cleared on the VCPU triple-fault reset path.Fixes db5e82496492 ("KVM: SVM: Virtualize and advertise support for ERAPS")Assisted-by Gemini:Gemini-NextSigned-off-by Jim Mattson <[email protected]>--- arch/x86/kvm/svm/svm.c | 5 +++++ 1 file changed, 5 insertions(+)arch/x86/kvm/svm/svm.c +0
-0arch/x86/kvm/svm/svm.c static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 4742
Quoted 7 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -4736,6 +4736,11 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (guest_cpuid_is_intel_compatible (vcpu))guest_cpu_cap_clear (vcpu,X86_FEATURE_V_VMSAVE_VMLOAD ); +if (guest_cpu_cap_has (vcpu,X86_FEATURE_ERAPS )) + svm->vmcb01.ptr->control.erap_ctl |=ERAP_CONTROL_ALLOW_LARGER_RAP ; +else + svm->vmcb01.ptr->control.erap_ctl &= ~ERAP_CONTROL_ALLOW_LARGER_RAP ;If we handle ERAPS here, can't we drop the code in
init_vmcb()? In the context of PML enablement, AMD is updating the APM to clarify that the control area remains valid after shutdown, i.e. KVM doesn't need to re-initializeerap_ctlwhen synthesizing INIT fromshutdown_interception().
Quoted 5 lines
"After an intercepted shutdown, the VMCB control area is valid (with the
exception of offsets 60h, 61h, and 68h) and the VMCB state save area is undefined."
LOL, nice, I just saw the Sashiko report from v1. I think the way to handle this is to go with this version, and then rework shutdown_interception() as part of PML enablement to not clear the entire page, only the contents that may be corrupted.
PATCH v2 2/3 KVM: SVM: Dirty ERAPS register on all ASID TLB flushes
In reply to Jim MattsonPer the AMD APM, ERAPS clears the Return Address Predictor (RAP/RSB) on all implicit TLB invalidations (e.g. modifying certain CR[04] bits).
Move kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS) into svm_flush_tlb_asid() so that any ASID-level TLB flush marks ERAPS dirty. Centralizing the ERAPS dirty call in svm_flush_tlb_asid() ensures that all ASID flushes (flush_tlb_current, flush_tlb_all, flush_tlb_guest) properly set ERAP_CONTROL_CLEAR_RAP on the next VMRUN.
Remove the now redundant svm_flush_tlb_guest() wrapper.
--- arch/x86/kvm/svm/svm.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/svm.c +3 -8
@@ -4171,6 +4171,8 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); +kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); +/* * Unlike VMX, SVM doesn't provide a way to flush only NPT TLB entries. * A TLB flush for the current ASID flushes both "host" and "guest" TLB
@@ -4229,13 +4231,6 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)invlpga (gva, svm->vmcb->control.asid); } -static void svm_flush_tlb_guest (struct kvm_vcpu *vcpu) -{ -kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); - -svm_flush_tlb_asid (vcpu); -} -static inline void sync_cr8_to_lapic (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -5383,7 +5378,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .flush_tlb_all = svm_flush_tlb_all, .flush_tlb_current = svm_flush_tlb_current, .flush_tlb_gva = svm_flush_tlb_gva, - .flush_tlb_guest = svm_flush_tlb_guest, + .flush_tlb_guest = svm_flush_tlb_asid, .vcpu_pre_run = svm_vcpu_pre_run, .vcpu_run = svm_vcpu_run,
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Re: [PATCH v2 2/3] KVM: SVM: Dirty ERAPS register on all ASID TLB flushes
Quoted 5 lines · Jim Mattson
Per the AMD APM, ERAPS clears the Return Address Predictor (RAP/RSB) on all implicit TLB invalidations (e.g. modifying certain CR[04] bits).
Move
kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS)intosvm_flush_tlb_asid()so that any ASID-level TLB flush marks ERAPS dirty.
NAK, the whole point of KVM_REQ_TLB_FLUSH_GUEST is emulate TLB invalidations in the guest's domain. If we're missing KVM_REQ_TLB_FLUSH_GUEST requests, fix those.
I think we're missing a clear on the MOV CR3 flush when TDP is disabled? Beyond that, nothing jumps out.
x86/kvm/x86.c +6 -12
@@ -755,10 +755,16 @@ void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid)* also via the emulator. KVM's TDP page tables are not in the scope of * the invalidation, but the guest's TLB entries need to be flushed as * the CPU may have cached entries in its TLB for the target PCID. +* +* When ERAPS is supported, invalidating a specific PCID clears the RAP +* (Return Address Predicator). KVM flushes the RAP when emulating a +* full guest TLB flush, so only the !TDP case needs to be handled here. */ if (unlikely (tdp_enabled)) {kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);return ; + }else if (guest_cpu_cap_has (vcpu,X86_FEATURE_ERAPS )) { +kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); }/*
@@ -10740,13 +10746,6 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva)return 1 ; } -/* -* When ERAPS is supported, invalidating a specific PCID clears -* the RAP (Return Address Predicator). -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_ERAPS )) -kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); -kvm_invalidate_pcid (vcpu, operand.pcid);return kvm_skip_emulated_instruction (vcpu);
@@ -10760,11 +10759,6 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) fallthrough;case INVPCID_TYPE_ALL_INCL_GLOBAL : -/* -* Don't bother marking VCPU_REG_ERAPS dirty, SVM will take -* care of doing so when emulating the full guest TLB flush -* (the RAP is cleared on all implicit TLB flushes). -*/ kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);return kvm_skip_emulated_instruction (vcpu);
Quoted 52 lines
Centralizing the ERAPS dirty call in
svm_flush_tlb_asid()ensures that all ASID flushes (flush_tlb_current,flush_tlb_all,flush_tlb_guest) properly setERAP_CONTROL_CLEAR_RAPon the next VMRUN.Remove the now redundant
svm_flush_tlb_guest()wrapper.Fixes db5e82496492 ("KVM: SVM: Virtualize and advertise support for ERAPS")Assisted-by Gemini:Gemini-NextSigned-off-by Jim Mattson <[email protected]>--- arch/x86/kvm/svm/svm.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)arch/x86/kvm/svm/svm.c +3
-8@@ -4171,6 +4171,8 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); +kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); +/* * Unlike VMX, SVM doesn't provide a way to flush only NPT TLB entries. * A TLB flush for the current ASID flushes both "host" and "guest" TLB @@ -4229,13 +4231,6 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)invlpga (gva, svm->vmcb->control.asid); } -static void svm_flush_tlb_guest (struct kvm_vcpu *vcpu) -{ -kvm_register_mark_dirty (vcpu,VCPU_REG_ERAPS ); - -svm_flush_tlb_asid (vcpu); -} -static inline void sync_cr8_to_lapic (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);@@ -5383,7 +5378,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .flush_tlb_all = svm_flush_tlb_all, .flush_tlb_current = svm_flush_tlb_current, .flush_tlb_gva = svm_flush_tlb_gva, - .flush_tlb_guest = svm_flush_tlb_guest, + .flush_tlb_guest = svm_flush_tlb_asid, .vcpu_pre_run = svm_vcpu_pre_run, .vcpu_run = svm_vcpu_run,Signature · 1 line
2.55.0.229.g6434b31f56-goog
Re: [PATCH v2 2/3] KVM: SVM: Dirty ERAPS register on all ASID TLB flushes
Quoted 5 lines · Sean Christopherson
Quoted 3 lines · Jim Mattson
Per the AMD APM, ERAPS clears the Return Address Predictor (RAP/RSB) on all implicit TLB invalidations (e.g. modifying certain CR[04] bits).
Quoted 8 lines
Quoted 2 lines
Move
kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS)intosvm_flush_tlb_asid()so that any ASID-level TLB flush marks ERAPS dirty.NAK, the whole point of
KVM_REQ_TLB_FLUSH_GUESTis emulate TLB invalidations in the guest's domain. If we're missingKVM_REQ_TLB_FLUSH_GUESTrequests, fix those.I think we're missing a clear on the MOV CR3 flush when TDP is disabled? Beyond that, nothing jumps out.
Yes, I think that's it.
PATCH v2 3/3 KVM: x86: Flush guest TLB on MTRR MSR writes
In reply to Jim MattsonPer both the AMD APM and the Intel SDM, writing to an MTRR with WRMSR is an implicit TLB invalidation that invalidates all TLB entries (including global entries). [The APM uses the word, "update," which could be construed to mean "modify," but it is unclear.]
Previously, kvm_mtrr_set_msr() stored the updated MTRR state without requesting a TLB flush. This broke x86 architectural compliance by leaving existing TLB entries in hardware. Additionally, on AMD CPUs supporting ERAPS, omitting the TLB flush request meant VCPU_REG_ERAPS was never dirtied on MTRR writes, leaving the Return Address Predictor (RAP/RSB) uncleared.
Issue a KVM_REQ_TLB_FLUSH_GUEST request on every MTRR MSR write to match x86 architectural semantics.
--- arch/x86/kvm/mtrr.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kvm/mtrr.c +1 -0
@@ -105,6 +105,7 @@ int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data)return 1 ; *mtrr = data; +kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);return 0 ; }
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Sean Christopherson
maintainerRe: [PATCH v2 3/3] KVM: x86: Flush guest TLB on MTRR MSR writes
Quoted 4 lines · Jim Mattson
Per both the AMD APM and the Intel SDM, writing to an MTRR with WRMSR is an implicit TLB invalidation that invalidates all TLB entries (including global entries). [The APM uses the word, "update," which could be construed to mean "modify," but it is unclear.]
Yeesh, the SDM isn't very helpful. Under the description of WRMSR, it very clearly says:
When the WRMSR instruction is used to write to an MTRR, the TLBs are invalidated.
But then in the recommended pseudocode, the SDM says software should flush TLBs:
pre_mtrr_change()
BEGIN
disable interrupts;
Save current value of CR4;
disable and flush caches;
flush TLBs;
disable MTRRs;
IF multiprocessing
THEN maintain consistency through IPIs;
FI;
END post_mtrr_change()
BEGIN
flush caches and TLBs;
enable MTRRs;
enable caches;
restore value of CR4;
enable interrupts;
ENDQuoted 6 lines
Previously,
kvm_mtrr_set_msr()stored the updated MTRR state without requesting a TLB flush. This broke x86 architectural compliance by leaving existing TLB entries in hardware. Additionally, on AMD CPUs supporting ERAPS, omitting the TLB flush request meantVCPU_REG_ERAPSwas never dirtied on MTRR writes, leaving the Return Address Predictor (RAP/RSB) uncleared.
Heh, Andy Cooper even pointed out the implicit flushes on MTRR writes[1], and no one noticed KVM was missing that specific flush. I mention that mostly because the context of that thread was asking AMD to drop the statement that the RAP is cleared on writes to "other model specific MSRs, see NDA docs", so that hypervisors wouldn't have to clear the RAP on every WRMSR.
AMD updated their documentation[2] to say that clears on MSR writes are considered microarchitectural, but this patch and the above blurb are still accurate because MTRR writes are covered by the "implicit TLB flush" clause, not the generic WRMSR clause.
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
Quoted 25 lines
Issue a
KVM_REQ_TLB_FLUSH_GUESTrequest on every MTRR MSR write to match x86 architectural semantics.Fixes 9ba075a664df ("KVM: MTRR support")Assisted-by Gemini:Gemini-NextSigned-off-by Jim Mattson <[email protected]>--- arch/x86/kvm/mtrr.c | 1 + 1 file changed, 1 insertion(+)arch/x86/kvm/mtrr.c +1
-0@@ -105,6 +105,7 @@ int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data)return 1 ; *mtrr = data; +kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);return 0 ; }Signature · 1 line
2.55.0.229.g6434b31f56-goog