KVM: SVM: Refactor vcpu_load/put to use vmload/vmsave for host state
cover KVM: SVM: Refactor vcpu_load/put to use vmload/vmsave for host state
This series re-works the SVM KVM implementation to use vmload/vmsave to handle saving/restoring additional host MSRs rather than explicit MSR read/writes, resulting in a significant performance improvement for some specific workloads and simplifying some of the save/load code (PATCH 1).
With those changes some commonalities emerge between SEV-ES and normal vcpu_load/vcpu_put paths, which we then take advantage of to share more code, as well as refactor them in a way that more closely aligns with the VMX implementation (PATCH 2 and 3).
v3:
- rebased on kvm-next
- remove uneeded braces from host MSR save/load loops (Sean)
- use page_to_phys() in place of page_to_pfn() and shifting (Sean)
- use stack instead of struct field to cache host save area outside of per-cpu storage, and pass as an argument to __svm_vcpu_run() to handle the VMLOAD in ASM code rather than inlining ASM (Sean/Andy)
- remove now-uneeded index/sev_es_restored fields from host_save_user_msrs list
- move host-saving/guest-loading of registers to prepare_guest_switch(), and host-loading of registers to prepare_host_switch, for both normal and sev-es paths (Sean)
v2:
- rebase on latest kvm/next
- move VMLOAD to just after vmexit so we can use it to handle all FS/GS host state restoration and rather than relying on loadsegment() and explicit write to MSR_GS_BASE (Andy)
- drop 'host' field from struct vcpu_svm since it is no longer needed for storing FS/GS/LDT state (Andy)
arch/x86/kvm/svm/sev.c | 30 +----------------------------- arch/x86/kvm/svm/svm.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------ arch/x86/kvm/svm/svm.h | 31 ++++++------------------------- arch/x86/kvm/svm/vmenter.S | 10 ++++++++++ 4 files changed, 75 insertions(+), 108 deletions(-)
PATCH v3 1/3 KVM: SVM: use vmsave/vmload for saving/restoring additional host state
Using a guest workload which simply issues 'hlt' in a tight loop to generate VMEXITs, it was observed (on a recent EPYC processor) that a significant amount of the VMEXIT overhead measured on the host was the result of MSR reads/writes in svm_vcpu_load/svm_vcpu_put according to perf:
67.49%--kvm_arch_vcpu_ioctl_run
|
|--23.13%--vcpu_put
| kvm_arch_vcpu_put
| |
| |--21.31%--native_write_msr
| |
| --1.27%--svm_set_cr4
|
|--16.11%--vcpu_load
| |
| --15.58%--kvm_arch_vcpu_load
| |
| |--13.97%--svm_set_cr4
| | |
| | |--12.64%--native_read_msrMost of these MSRs relate to 'syscall'/'sysenter' and segment bases, and can be saved/restored using 'vmsave'/'vmload' instructions rather than explicit MSR reads/writes. In doing so there is a significant reduction in the svm_vcpu_load/svm_vcpu_put overhead measured for the above workload:
50.92%--kvm_arch_vcpu_ioctl_run
|
|--19.28%--disable_nmi_singlestep
|
|--13.68%--vcpu_load
| kvm_arch_vcpu_load
| |
| |--9.19%--svm_set_cr4
| | |
| | --6.44%--native_read_msr
| |
| --3.55%--native_write_msr
|
|--6.05%--kvm_inject_nmi
|--2.80%--kvm_sev_es_mmio_read
|--2.19%--vcpu_put
| |
| --1.25%--kvm_arch_vcpu_put
| native_write_msrQuantifying this further, if we look at the raw cycle counts for a normal iteration of the above workload (according to 'rdtscp'), kvm_arch_vcpu_ioctl_run() takes ~4600 cycles from start to finish with the current behavior. Using 'vmsave'/'vmload', this is reduced to ~2800 cycles, a savings of 39%.
While this approach doesn't seem to manifest in any noticeable improvement for more realistic workloads like UnixBench, netperf, and kernel builds, likely due to their exit paths generally involving IO with comparatively high latencies, it does improve overall overhead of KVM_RUN significantly, which may still be noticeable for certain situations. It also simplifies some aspects of the code.
With this change, explicit save/restore is no longer needed for the following host MSRs, since they are documented[1] as being part of the VMCB State Save Area:
MSR_STAR, MSR_LSTAR, MSR_CSTAR,
MSR_SYSCALL_MASK, MSR_KERNEL_GS_BASE,
MSR_IA32_SYSENTER_CS,
MSR_IA32_SYSENTER_ESP,
MSR_IA32_SYSENTER_EIP,
MSR_FS_BASE, MSR_GS_BASE
and only the following MSR needs individual handling in svm_vcpu_put/svm_vcpu_load:
MSR_TSC_AUX
We could drop the host_save_user_msrs array/loop and instead handle MSR read/write of MSR_TSC_AUX directly, but we leave that for now as a potential follow-up.
Since 'vmsave'/'vmload' also handles the LDTR and FS/GS segment registers (and associated hidden state)[2], some of the code previously used to handle this is no longer needed, so we drop it as well.
The first public release of the SVM spec[3] also documents the same handling for the host state in question, so we make these changes unconditionally.
Also worth noting is that we 'vmsave' to the same page that is subsequently used by 'vmrun' to record some host additional state. This is okay, since, in accordance with the spec[2], the additional state written to the page by 'vmrun' does not overwrite any fields written by 'vmsave'. This has also been confirmed through testing (for the above CPU, at least).
[1] AMD64 Architecture Programmer's Manual, Rev 3.33, Volume 2, Appendix B, Table B-2
[2] AMD64 Architecture Programmer's Manual, Rev 3.31, Volume 3, Chapter 4, VMSAVE/VMLOAD
[3] Secure Virtual Machine Architecture Reference Manual, Rev 3.01
--- arch/x86/kvm/svm/svm.c | 36 +++++++----------------------------- arch/x86/kvm/svm/svm.h | 19 +------------------ arch/x86/kvm/svm/vmenter.S | 10 ++++++++++ 3 files changed, 18 insertions(+), 47 deletions(-)
arch/x86/kvm/svm/svm.c +7 -29
@@ -1420,16 +1420,12 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (sev_es_guest (svm->vcpu.kvm)) {sev_es_vcpu_load (svm, cpu); }else { -#ifdef CONFIG_X86_64 -rdmsrl (MSR_GS_BASE ,to_svm (vcpu)->host.gs_base); -#endif -savesegment (fs, svm->host.fs); -savesegment (gs, svm->host.gs); - svm->host.ldt =kvm_read_ldt (); -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)rdmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]); + + asm volatile(__ex("vmsave %%"_ASM_AX) + : : "a" (page_to_phys(sd->save_area)) : "memory"); }if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) {
@@ -1461,17 +1457,6 @@ static void svm_vcpu_put(struct kvm_vcpu *vcpu)if (sev_es_guest (svm->vcpu.kvm)) {sev_es_vcpu_put (svm); }else { -kvm_load_ldt (svm->host.ldt); -#ifdef CONFIG_X86_64 -loadsegment (fs, svm->host.fs); -wrmsrl (MSR_KERNEL_GS_BASE , current->thread.gsbase); -load_gs_index (svm->host.gs); -#else -#ifdef CONFIG_X86_32_LAZY_GS -loadsegment (gs, svm->host.gs); -#endif -#endif -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)wrmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]);
@@ -3675,7 +3660,7 @@ static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu)return EXIT_FASTPATH_NONE ; } -void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs); +void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs,unsigned long hostsa_pa); static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,struct vcpu_svm *svm)
@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,if (sev_es_guest (svm->vcpu.kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa); }else { -__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64 -native_wrmsrl (MSR_GS_BASE , svm->host.gs_base); -#else -loadsegment (fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS -loadsegment (gs, svm->host.gs); -#endif -#endif +__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, +page_to_phys (per_cpu (svm_data, + vcpu->cpu)->save_area)); }/*
arch/x86/kvm/svm/svm.h +1 -18
@@ -27,17 +27,6 @@ static const struct svm_host_save_msrs {u32 index;/* Index of the MSR */ bool sev_es_restored;/* True if MSR is restored on SEV-ES VMEXIT */ } host_save_user_msrs[] = { -#ifdef CONFIG_X86_64 - { .index = MSR_STAR, .sev_es_restored = true }, - { .index = MSR_LSTAR, .sev_es_restored = true }, - { .index = MSR_CSTAR, .sev_es_restored = true }, - { .index = MSR_SYSCALL_MASK, .sev_es_restored = true }, - { .index = MSR_KERNEL_GS_BASE, .sev_es_restored = true }, - { .index = MSR_FS_BASE, .sev_es_restored = true }, -#endif - { .index = MSR_IA32_SYSENTER_CS, .sev_es_restored = true }, - { .index = MSR_IA32_SYSENTER_ESP, .sev_es_restored = true }, - { .index = MSR_IA32_SYSENTER_EIP, .sev_es_restored = true }, { .index = MSR_TSC_AUX, .sev_es_restored = false }, }; #define NR_HOST_SAVE_USER_MSRS ARRAY_SIZE(host_save_user_msrs)
@@ -130,12 +119,6 @@ struct vcpu_svm {u64 next_rip;u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS ]; -struct { -u16 fs; -u16 gs; -u16 ldt; -u64 gs_base; - } host;u64 spec_ctrl;/*
@@ -595,6 +578,6 @@ void sev_es_vcpu_put(struct vcpu_svm *svm);/* vmenter.S */ void __svm_sev_es_vcpu_run (unsigned long vmcb_pa); -void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs); +void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs,unsigned long hostsa_pa);#endif
arch/x86/kvm/svm/vmenter.S +10 -0
@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP
@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endif push %_ASM_BX + /* Save @hostsa_pa */ + push %_ASM_ARG3 + /* Save @regs. */ push %_ASM_ARG2
@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif + /* "POP" @hostsa_pa to RAX. */ + pop %_ASM_AX + + /* Restore host user state and FS/GS base */ + vmload %_ASM_AX + pop %_ASM_BX #ifdef CONFIG_X86_64
Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
Quoted 7 lines · Michael Roth
arch/x86/kvm/svm/svm.c +0
-3@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,if (sev_es_guest (svm->vcpu.kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa); }else { -__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64
arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, · line 3693
Quoted 10 lines
arch/x86/kvm/svm/svm.c +3
-7-native_wrmsrl (MSR_GS_BASE , svm->host.gs_base); -#else -loadsegment (fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS -loadsegment (gs, svm->host.gs); -#endif -#endif +__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, +page_to_phys (per_cpu (svm_data, + vcpu->cpu)->save_area));
Does this need to use __sme_page_pa()?
Quoted 3 lines
(quoted diff) +0
-0} /*
...
Quoted 23 lines
arch/x86/kvm/svm/vmenter.S +4
-0@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endif push %_ASM_BX + /* Save @hostsa_pa */ + push %_ASM_ARG3 + /* Save @regs. */ push %_ASM_ARG2
arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_vcpu_run) · line 165
Quoted 8 lines
arch/x86/kvm/svm/vmenter.S +5
-0@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif + /* "POP" @hostsa_pa to RAX. */ + pop %_ASM_AX + + /* Restore host user state and FS/GS base */ + vmload %_ASM_AX
This VMLOAD needs the "handle fault on reboot" goo. Seeing the code, I think I'd prefer to handle this in C code, especially if Paolo takes the svm_ops.h patch[*]. Actually, I think with that patch it'd make sense to move the existing VMSAVE+VMLOAD for the guest into svm.c, too. And completely unrelated, the fault handling in svm/vmenter.S can be cleaned up a smidge to eliminate the JMPs.
Paolo, what do you think about me folding these patches into my series to do the above cleanups? And maybe sending a pull request for the end result? (I'd also like to add on a patch to use the user return MSR mechanism for MSR_TSC_AUX).
Quoted 7 lines
arch/x86/kvm/svm/vmenter.S +1
-0+ pop %_ASM_BX #ifdef CONFIG_X86_64Signature · 1 line
2.25.1
Re: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · Michael Roth
arch/x86/kvm/svm/svm.c +0
-7@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,if (sev_es_guest (svm->vcpu.kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa); }else { -__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64 -native_wrmsrl (MSR_GS_BASE , svm->host.gs_base); -#else -loadsegment (fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS
Quoted 8 lines
arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, · line 3693
Quoted 6 lines
arch/x86/kvm/svm/svm.c +3
-3-loadsegment (gs, svm->host.gs); -#endif -#endif +__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, +page_to_phys (per_cpu (svm_data, + vcpu->cpu)->save_area));Does this need to use
__sme_page_pa()?
Yes, it should now. The SEV-ES support added the SME encryption bit to the MSR_VM_HSAVE_PA MSR, so we need to be consistent in how the data is read and written.
Thanks,
Tom
Quoted 59 lines
Quoted 4 lines
(quoted diff) +0
-0} /*...
Quoted 23 lines
arch/x86/kvm/svm/vmenter.S +4
-0@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endif push %_ASM_BX + /* Save @hostsa_pa */ + push %_ASM_ARG3 + /* Save @regs. */ push %_ASM_ARG2arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_vcpu_run) · line 165
Quoted 8 lines
arch/x86/kvm/svm/vmenter.S +5
-0@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif + /* "POP" @hostsa_pa to RAX. */ + pop %_ASM_AX + + /* Restore host user state and FS/GS base */ + vmload %_ASM_AXThis VMLOAD needs the "handle fault on reboot" goo. Seeing the code, I think I'd prefer to handle this in C code, especially if Paolo takes the
svm_ops.hpatch[*]. Actually, I think with that patch it'd make sense to move the existing VMSAVE+VMLOAD for the guest intosvm.c, too. And completely unrelated, the fault handling insvm/vmenter.Scan be cleaned up a smidge to eliminate the JMPs.Paolo, what do you think about me folding these patches into my series to do the above cleanups? And maybe sending a pull request for the end result? (I'd also like to add on a patch to use the user return MSR mechanism for
MSR_TSC_AUX).Quoted 7 lines
arch/x86/kvm/svm/vmenter.S +1
-0+ pop %_ASM_BX #ifdef CONFIG_X86_64Signature · 1 line
2.25.1
Re: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
Quoted 25 lines · Tom Lendacky
Quoted 15 lines · Sean Christopherson
Quoted 14 lines · Michael Roth
arch/x86/kvm/svm/svm.c +0
-8@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(structkvm_vcpu *vcpu, if (sev_es_guest(svm->vcpu.kvm)) { __svm_sev_es_vcpu_run(svm->vmcb_pa); } else { - __svm_vcpu_run(svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64 - native_wrmsrl(MSR_GS_BASE, svm->host.gs_base); -#else - loadsegment(fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS - loadsegment(gs, svm->host.gs);Quoted 8 lines
arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, · line 3693
Quoted 6 lines
arch/x86/kvm/svm/svm.c +3
-2-#endif -#endif + __svm_vcpu_run(svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, + page_to_phys(per_cpu(svm_data, + vcpu->cpu)->save_area));Does this need to use
__sme_page_pa()?
Quoted 1 line
Yes, it should now. The SEV-ES support added the SME encryption bit to the
MSR_VM_HSAVE_PAMSR, so we need to be consistent in how the data is read and written.
Oh, and also in svm_vcpu_load().
Thanks,
Tom
Quoted 58 lines
Quoted 1 line
Thanks,
Tom
Quoted 55 lines
Quoted 3 lines
}
/*...
Quoted 21 lines
arch/x86/kvm/svm/vmenter.S +1
-0@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run)push %_ASM_BP
arch/x86/kvm/svm/vmenter.S +0
-0@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endifpush %_ASM_BX
+ /* Save @hostsa_pa */
+ push %_ASM_ARG3
+
/* Save @regs. */
push %_ASM_ARG2arch/x86/kvm/svm/vmenter.S +0
-0@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run)Quoted 7 lines
xor %r15d, %r15d
#endif
+ /* "POP" @hostsa_pa to RAX. */
+ pop %_ASM_AX
+
+ /* Restore host user state and FS/GS base */
+ vmload %_ASM_AXThis VMLOAD needs the "handle fault on reboot" goo. Seeing the code, I think I'd prefer to handle this in C code, especially if Paolo takes the
svm_ops.hpatch[*]. Actually, I think with that patch it'd make sense to move the existing VMSAVE+VMLOAD for the guest intosvm.c, too. And completely unrelated, the fault handling insvm/vmenter.Scan be cleaned up a smidge to eliminate the JMPs.Paolo, what do you think about me folding these patches into my series to do the above cleanups? And maybe sending a pull request for the end result? (I'd also like to add on a patch to use the user return MSR mechanism for
MSR_TSC_AUX).Quoted 6 lines
+
pop %_ASM_BX
#ifdefCONFIG_X86_64Signature · 1 line
2.25.1
Re: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
In reply to Sean ChristophersonQuoted 12 lines · Sean Christopherson
Quoted 11 lines · Michael Roth
arch/x86/kvm/svm/svm.c +0
-7@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,if (sev_es_guest (svm->vcpu.kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa); }else { -__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64 -native_wrmsrl (MSR_GS_BASE , svm->host.gs_base); -#else -loadsegment (fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS
Quoted 8 lines
arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, · line 3693
Quoted 6 lines
arch/x86/kvm/svm/svm.c +3
-3-loadsegment (gs, svm->host.gs); -#endif -#endif +__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, +page_to_phys (per_cpu (svm_data, + vcpu->cpu)->save_area));Does this need to use
__sme_page_pa()?
Oddly enough the current patch seems to work even with SME enabled. Not sure why though since as Tom pointed out we do use it elsewhere with the SME bit set. But should be setting it either way.
Quoted 33 lines
Quoted 4 lines
(quoted diff) +0
-0} /*...
Quoted 26 lines
arch/x86/kvm/svm/vmenter.S +4
-0@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endif push %_ASM_BX + /* Save @hostsa_pa */ + push %_ASM_ARG3 + /* Save @regs. */ push %_ASM_ARG2@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif
Quoted 7 lines
arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_vcpu_run) · line 165
Quoted 5 lines
arch/x86/kvm/svm/vmenter.S +5
-0+ /* "POP" @hostsa_pa to RAX. */ + pop %_ASM_AX + + /* Restore host user state and FS/GS base */ + vmload %_ASM_AXThis VMLOAD needs the "handle fault on reboot" goo. Seeing the code, I think
Ah, yes, I overlooked that with the rework.
Quoted 9 lines
I'd prefer to handle this in C code, especially if Paolo takes the
svm_ops.hpatch[*]. Actually, I think with that patch it'd make sense to move the existing VMSAVE+VMLOAD for the guest intosvm.c, too. And completely unrelated, the fault handling insvm/vmenter.Scan be cleaned up a smidge to eliminate the JMPs.Paolo, what do you think about me folding these patches into my series to do the above cleanups? And maybe sending a pull request for the end result? (I'd also like to add on a patch to use the user return MSR mechanism for
MSR_TSC_AUX).
No complaints on my end at least :) But happy to send a v4 with SME bit fix and reboot handling if you think that's worthwhile (and other suggested changes as well, but not sure exactly what you have in mind there). Can also help with any testing of course.
Thanks,
Mike
Quoted 10 lines
Quoted 7 lines
arch/x86/kvm/svm/vmenter.S +1
-0+ pop %_ASM_BX #ifdef CONFIG_X86_64Signature · 1 line
2.25.1
Paolo Bonzini
maintainerRe: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
In reply to Sean ChristophersonQuoted 10 lines · Sean Christopherson
This VMLOAD needs the "handle fault on reboot" goo. Seeing the code, I think I'd prefer to handle this in C code, especially if Paolo takes the
svm_ops.hpatch[*]. Actually, I think with that patch it'd make sense to move the existing VMSAVE+VMLOAD for the guest intosvm.c, too. And completely unrelated, the fault handling insvm/vmenter.Scan be cleaned up a smidge to eliminate the JMPs.Paolo, what do you think about me folding these patches into my series to do the above cleanups? And maybe sending a pull request for the end result? (I'd also like to add on a patch to use the user return MSR mechanism for
MSR_TSC_AUX).
I have queued that part already, so Mike can rebase on top of kvm/queue.
Paolo
Quoted 2 lines
Re: [PATCH v3 1/3] KVM: SVM: use vmsave/vmload for saving/restoring additional host state
In reply to Michael RothQuoted 153 lines · Michael Roth
Using a guest workload which simply issues 'hlt' in a tight loop to generate VMEXITs, it was observed (on a recent EPYC processor) that a significant amount of the VMEXIT overhead measured on the host was the result of MSR reads/writes in svm_vcpu_load/svm_vcpu_put according to perf:
67.49%--kvm_arch_vcpu_ioctl_run | |--23.13%--vcpu_put | kvm_arch_vcpu_put | | | |--21.31%--native_write_msr | | | --1.27%--svm_set_cr4 | |--16.11%--vcpu_load | | | --15.58%--kvm_arch_vcpu_load | | | |--13.97%--svm_set_cr4 | | | | | |--12.64%--native_read_msrMost of these MSRs relate to 'syscall'/'sysenter' and segment bases, and can be saved/restored using 'vmsave'/'vmload' instructions rather than explicit MSR reads/writes. In doing so there is a significant reduction in the svm_vcpu_load/svm_vcpu_put overhead measured for the above workload:
50.92%--kvm_arch_vcpu_ioctl_run | |--19.28%--disable_nmi_singlestep | |--13.68%--vcpu_load | kvm_arch_vcpu_load | | | |--9.19%--svm_set_cr4 | | | | | --6.44%--native_read_msr | | | --3.55%--native_write_msr | |--6.05%--kvm_inject_nmi |--2.80%--kvm_sev_es_mmio_read |--2.19%--vcpu_put | | | --1.25%--kvm_arch_vcpu_put | native_write_msrQuantifying this further, if we look at the raw cycle counts for a normal iteration of the above workload (according to 'rdtscp'),
kvm_arch_vcpu_ioctl_run()takes ~4600 cycles from start to finish with the current behavior. Using 'vmsave'/'vmload', this is reduced to ~2800 cycles, a savings of 39%.While this approach doesn't seem to manifest in any noticeable improvement for more realistic workloads like UnixBench, netperf, and kernel builds, likely due to their exit paths generally involving IO with comparatively high latencies, it does improve overall overhead of
KVM_RUNsignificantly, which may still be noticeable for certain situations. It also simplifies some aspects of the code.With this change, explicit save/restore is no longer needed for the following host MSRs, since they are documented[1] as being part of the VMCB State Save Area:
MSR_STAR, MSR_LSTAR, MSR_CSTAR, MSR_SYSCALL_MASK, MSR_KERNEL_GS_BASE, MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, MSR_FS_BASE, MSR_GS_BASEand only the following MSR needs individual handling in svm_vcpu_put/svm_vcpu_load:
MSR_TSC_AUXWe could drop the
host_save_user_msrsarray/loop and instead handle MSR read/write ofMSR_TSC_AUXdirectly, but we leave that for now as a potential follow-up.Since 'vmsave'/'vmload' also handles the LDTR and FS/GS segment registers (and associated hidden state)[2], some of the code previously used to handle this is no longer needed, so we drop it as well.
The first public release of the SVM spec[3] also documents the same handling for the host state in question, so we make these changes unconditionally.
Also worth noting is that we 'vmsave' to the same page that is subsequently used by 'vmrun' to record some host additional state. This is okay, since, in accordance with the spec[2], the additional state written to the page by 'vmrun' does not overwrite any fields written by 'vmsave'. This has also been confirmed through testing (for the above CPU, at least).
[1] AMD64 Architecture Programmer's Manual, Rev 3.33, Volume 2, Appendix B, Table B-2
[2] AMD64 Architecture Programmer's Manual, Rev 3.31, Volume 3, Chapter 4, VMSAVE/VMLOAD
[3] Secure Virtual Machine Architecture Reference Manual, Rev 3.01Suggested-by Tom Lendacky <[email protected]>Signed-off-by Michael Roth <[email protected]>--- arch/x86/kvm/svm/svm.c | 36 +++++++----------------------------- arch/x86/kvm/svm/svm.h | 19 +------------------ arch/x86/kvm/svm/vmenter.S | 10 ++++++++++ 3 files changed, 18 insertions(+), 47 deletions(-)arch/x86/kvm/svm/svm.c +3
-18@@ -1420,16 +1420,12 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (sev_es_guest (svm->vcpu.kvm)) {sev_es_vcpu_load (svm, cpu); }else { -#ifdef CONFIG_X86_64 -rdmsrl (MSR_GS_BASE ,to_svm (vcpu)->host.gs_base); -#endif -savesegment (fs, svm->host.fs); -savesegment (gs, svm->host.gs); - svm->host.ldt =kvm_read_ldt (); -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)rdmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]); + + asm volatile(__ex("vmsave %%"_ASM_AX) + : : "a" (page_to_phys(sd->save_area)) : "memory"); }if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) {@@ -1461,17 +1457,6 @@ static void svm_vcpu_put(struct kvm_vcpu *vcpu)if (sev_es_guest (svm->vcpu.kvm)) {sev_es_vcpu_put (svm); }else { -kvm_load_ldt (svm->host.ldt); -#ifdef CONFIG_X86_64 -loadsegment (fs, svm->host.fs); -wrmsrl (MSR_KERNEL_GS_BASE , current->thread.gsbase); -load_gs_index (svm->host.gs); -#else -#ifdef CONFIG_X86_32_LAZY_GS -loadsegment (gs, svm->host.gs); -#endif -#endif -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)wrmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]);
arch/x86/kvm/svm/svm.c static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu) · line 3663
Quoted 5 lines
arch/x86/kvm/svm/svm.c +1
-1@@ -3675,7 +3660,7 @@ static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu)return EXIT_FASTPATH_NONE ; } -void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs); +void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs,unsigned long hostsa_pa);
There was a follow on fix patch to remove this forward declaration since, for SEV-ES, I had moved it into svm.h without deleting it here. I'm not sure when it will hit Paolo's tree.
Thanks,
Tom
Quoted 102 lines
arch/x86/kvm/svm/svm.c +3
-10static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,struct vcpu_svm *svm)@@ -3703,16 +3688,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu,if (sev_es_guest (svm->vcpu.kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa); }else { -__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs); - -#ifdef CONFIG_X86_64 -native_wrmsrl (MSR_GS_BASE , svm->host.gs_base); -#else -loadsegment (fs, svm->host.fs); -#ifndef CONFIG_X86_32_LAZY_GS -loadsegment (gs, svm->host.gs); -#endif -#endif +__svm_vcpu_run (svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs, +page_to_phys (per_cpu (svm_data, + vcpu->cpu)->save_area)); }/* arch/x86/kvm/svm/svm.h +1
-18@@ -27,17 +27,6 @@ static const struct svm_host_save_msrs {u32 index;/* Index of the MSR */ bool sev_es_restored;/* True if MSR is restored on SEV-ES VMEXIT */ } host_save_user_msrs[] = { -#ifdef CONFIG_X86_64 - { .index = MSR_STAR, .sev_es_restored = true }, - { .index = MSR_LSTAR, .sev_es_restored = true }, - { .index = MSR_CSTAR, .sev_es_restored = true }, - { .index = MSR_SYSCALL_MASK, .sev_es_restored = true }, - { .index = MSR_KERNEL_GS_BASE, .sev_es_restored = true }, - { .index = MSR_FS_BASE, .sev_es_restored = true }, -#endif - { .index = MSR_IA32_SYSENTER_CS, .sev_es_restored = true }, - { .index = MSR_IA32_SYSENTER_ESP, .sev_es_restored = true }, - { .index = MSR_IA32_SYSENTER_EIP, .sev_es_restored = true }, { .index = MSR_TSC_AUX, .sev_es_restored = false }, }; #define NR_HOST_SAVE_USER_MSRS ARRAY_SIZE(host_save_user_msrs)@@ -130,12 +119,6 @@ struct vcpu_svm {u64 next_rip;u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS ]; -struct { -u16 fs; -u16 gs; -u16 ldt; -u64 gs_base; - } host;u64 spec_ctrl;/* @@ -595,6 +578,6 @@ void sev_es_vcpu_put(struct vcpu_svm *svm);/* vmenter.S */ void __svm_sev_es_vcpu_run (unsigned long vmcb_pa); -void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs); +void __svm_vcpu_run (unsigned long vmcb_pa,unsigned long *regs,unsigned long hostsa_pa);#endif arch/x86/kvm/svm/vmenter.S +10
-0@@ -33,6 +33,7 @@ * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode * @vmcb_pa: unsigned long * @regs: unsigned long * (to guest registers) + * @hostsa_pa: unsigned long */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP@@ -47,6 +48,9 @@ SYM_FUNC_START(__svm_vcpu_run) #endif push %_ASM_BX + /* Save @hostsa_pa */ + push %_ASM_ARG3 + /* Save @regs. */ push %_ASM_ARG2@@ -154,6 +158,12 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif + /* "POP" @hostsa_pa to RAX. */ + pop %_ASM_AX + + /* Restore host user state and FS/GS base */ + vmload %_ASM_AX + pop %_ASM_BX #ifdef CONFIG_X86_64
PATCH v3 2/3 KVM: SVM: remove uneeded fields from host_save_users_msrs
In reply to Michael RothNow that the set of host user MSRs that need to be individually saved/restored are the same with/without SEV-ES, we can drop the .sev_es_restored flag and just iterate through the list unconditionally for both cases. A subsequent patch can then move these loops to a common path.
--- arch/x86/kvm/svm/sev.c | 16 ++++------------ arch/x86/kvm/svm/svm.c | 6 ++---- arch/x86/kvm/svm/svm.h | 7 ++----- 3 files changed, 8 insertions(+), 21 deletions(-)
arch/x86/kvm/svm/sev.c +4 -12
@@ -2007,12 +2007,8 @@ void sev_es_vcpu_load(struct vcpu_svm *svm, int cpu)* Certain MSRs are restored on VMEXIT, only save ones that aren't * restored. */ -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) { -if (host_save_user_msrs[i].sev_es_restored) -continue ; - -rdmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]); - } +for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) +rdmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]);/* XCR0 is restored on VMEXIT, save the current host value */ hostsa = (struct vmcb_save_area *)(page_address (sd->save_area) +0x400 );
@@ -2033,10 +2029,6 @@ void sev_es_vcpu_put(struct vcpu_svm *svm)* Certain MSRs are restored on VMEXIT and were saved with vmsave in * sev_es_vcpu_load() above. Only restore ones that weren't. */ -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) { -if (host_save_user_msrs[i].sev_es_restored) -continue ; - -wrmsrl (host_save_user_msrs[i].index, svm->host_user_msrs[i]); - } +for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) +wrmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); }
arch/x86/kvm/svm/svm.c +2 -4
@@ -1421,8 +1421,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)sev_es_vcpu_load (svm, cpu); }else {for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) -rdmsrl (host_save_user_msrs[i].index, - svm->host_user_msrs[i]); +rdmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); asm volatile(__ex("vmsave %%"_ASM_AX) : : "a" (page_to_phys(sd->save_area)) : "memory");
@@ -1458,8 +1457,7 @@ static void svm_vcpu_put(struct kvm_vcpu *vcpu)sev_es_vcpu_put (svm); }else {for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) -wrmsrl (host_save_user_msrs[i].index, - svm->host_user_msrs[i]); +wrmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); } }
arch/x86/kvm/svm/svm.h +2 -5
@@ -23,11 +23,8 @@#define __sme_page_pa (x) __sme_set(page_to_pfn(x) << PAGE_SHIFT) -static const struct svm_host_save_msrs { -u32 index;/* Index of the MSR */ -bool sev_es_restored;/* True if MSR is restored on SEV-ES VMEXIT */ -} host_save_user_msrs[] = { - { .index =MSR_TSC_AUX , .sev_es_restored = false }, +static const u32 host_save_user_msrs[] = { +MSR_TSC_AUX , };#define NR_HOST_SAVE_USER_MSRS ARRAY_SIZE(host_save_user_msrs)
Signature · 1 line
2.25.1
PATCH v3 3/3 KVM: SVM: use .prepare_guest_switch() to handle CPU register save/setup
In reply to Michael RothCurrently we save host state like user-visible host MSRs, and do some initial guest register setup for MSR_TSC_AUX and MSR_AMD64_TSC_RATIO in svm_vcpu_load(). Defer this until just before we enter the guest by moving the handling to kvm_x86_ops.prepare_guest_switch() similarly to how it is done for the VMX implementation.
Additionally, since handling of saving/restoring host user MSRs is the same both with/without SEV-ES enabled, move that handling to common code.
--- arch/x86/kvm/svm/sev.c | 22 +----------- arch/x86/kvm/svm/svm.c | 76 +++++++++++++++++++++++++++++------------- arch/x86/kvm/svm/svm.h | 5 +-- 3 files changed, 56 insertions(+), 47 deletions(-)
arch/x86/kvm/svm/sev.c +1 -21
@@ -1990,11 +1990,10 @@ void sev_es_create_vcpu(struct vcpu_svm *svm) sev_enc_bit)); } -void sev_es_vcpu_load (struct vcpu_svm *svm,int cpu) +void sev_es_prepare_guest_switch (struct vcpu_svm *svm,unsigned int cpu) {struct svm_cpu_data *sd =per_cpu (svm_data, cpu);struct vmcb_save_area *hostsa; -unsigned int i;/* * As an SEV-ES guest, hardware will restore the host state on VMEXIT,
@@ -2003,13 +2002,6 @@ void sev_es_vcpu_load(struct vcpu_svm *svm, int cpu)*/ asm volatile(__ex("vmsave") : : "a" (__sme_page_pa(sd->save_area)) : "memory"); -/* -* Certain MSRs are restored on VMEXIT, only save ones that aren't -* restored. - */ - for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) -rdmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); - /* XCR0 is restored on VMEXIT, save the current host value */ hostsa = (struct vmcb_save_area *)(page_address(sd->save_area) + 0x400); hostsa->xcr0 =xgetbv (XCR_XFEATURE_ENABLED_MASK );
@@ -2020,15 +2012,3 @@ void sev_es_vcpu_load(struct vcpu_svm *svm, int cpu)/* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */ hostsa->xss = host_xss; } - -void sev_es_vcpu_put (struct vcpu_svm *svm) -{ -unsigned int i; - -/* -* Certain MSRs are restored on VMEXIT and were saved with vmsave in -* sev_es_vcpu_load() above. Only restore ones that weren't. -*/ -for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++) -wrmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); -}
arch/x86/kvm/svm/svm.c +52 -24
119 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +3 -2
@@ -171,6 +171,8 @@ struct vcpu_svm {u64 ghcb_sa_len;bool ghcb_sa_sync;bool ghcb_sa_free; + +bool guest_state_loaded; };struct svm_cpu_data {
@@ -569,8 +571,7 @@ int sev_handle_vmgexit(struct vcpu_svm *svm);int sev_es_string_io (struct vcpu_svm *svm,int size,unsigned int port,int in);void sev_es_init_vmcb (struct vcpu_svm *svm);void sev_es_create_vcpu (struct vcpu_svm *svm); -void sev_es_vcpu_load (struct vcpu_svm *svm,int cpu); -void sev_es_vcpu_put (struct vcpu_svm *svm); +void sev_es_prepare_guest_switch (struct vcpu_svm *svm,unsigned int cpu);/* vmenter.S */
Signature · 1 line
2.25.1