KVM: nVMX: Optimize nested VM-Entry
cover KVM: nVMX: Optimize nested VM-Entry
The majority of patches in this series are loosely related optimizations to pick off low(ish) hanging fruit in nested VM-Entry, e.g. there are many VMREADs and VMWRITEs that can be optimized away without too much effort.
The major change (in terms of performance) is to not "put" the vCPU state when switching between vmcs01 and vmcs02, which can reudce the latency of a nested VM-Entry by upwards of 1000 cycles.
A few bug fixes are prepended as they touch code that happens to be modified by the various optimizations.
Sean Christopherson (15):
KVM: nVMX: Don't dump VMCS if virtual APIC page can't be mapped
KVM: VMX: Always signal #GP on WRMSR to MSR_IA32_CR_PAT with bad value
KVM: nVMX: Always sync GUEST_BNDCFGS when it comes from vmcs01
KVM: nVMX: Write ENCLS-exiting bitmap once per vmcs02
KVM: nVMX: Don't rewrite GUEST_PML_INDEX during nested VM-Entry
KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
KVM: nVMX: Don't reread VMCS-agnostic state when switching VMCS
KVM: nVMX: Don't speculatively write virtual-APIC page address
KVM: nVMX: Don't speculatively write APIC-access page address
KVM: nVMX: Update vmcs12 for MSR_IA32_CR_PAT when it's written
KVM: nVMX: Update vmcs12 for SYSENTER MSRs when they're written
KVM: nVMX: Update vmcs12 for MSR_IA32_DEBUGCTLMSR when it's written
KVM: nVMX: Update vmcs02 GUEST_IA32_DEBUGCTL only when vmcs12 is dirty
KVM: nVMX: Don't update GUEST_BNDCFGS if it's clean in HV eVMCS
KVM: nVMX: Copy PDPTRs to/from vmcs12 only when necessary
arch/x86/kvm/vmx/nested.c | 142 +++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.c | 93 +++++++++++++++++-------- arch/x86/kvm/vmx/vmx.h | 5 +- 3 files changed, 136 insertions(+), 104 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/15 KVM: nVMX: Don't dump VMCS if virtual APIC page can't be mapped
... as a malicious userspace can run a toy guest to generate invalid virtual-APIC page addresses in L1, i.e. flood the kernel log with error messages.
--- arch/x86/kvm/vmx/nested.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -3
@@ -2875,9 +2875,6 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)*/ vmcs_clear_bits (CPU_BASED_VM_EXEC_CONTROL ,CPU_BASED_TPR_SHADOW ); - }else { -printk ("bad virtual-APIC page address\n" ); -dump_vmcs (); } }
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 01/15] KVM: nVMX: Don't dump VMCS if virtual APIC page can't be mapped
Quoted 8 lines · Sean Christopherson
... as a malicious userspace can run a toy guest to generate invalid virtual-APIC page addresses in L1, i.e. flood the kernel log with error messages.
Fixes 690908104e39d ("KVM: nVMX: allow tests to use bad virtual-APIC page address")Cc Paolo Bonzini <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
The same is true even of dump_vmcs caused by emulation failures. I'm thinking of just hiding dump_vmcs beneath a module parameter.
Paolo
PATCH 2/15 KVM: VMX: Always signal #GP on WRMSR to MSR_IA32_CR_PAT with bad value
In reply to Sean ChristophersonThe behavior of WRMSR is in no way dependent on whether or not KVM consumes the value.
--- arch/x86/kvm/vmx/vmx.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +3 -2
@@ -1896,9 +1896,10 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) MSR_TYPE_W);break ;case MSR_IA32_CR_PAT : +if (!kvm_pat_valid (data)) +return 1 ; +if (vmcs_config.vmentry_ctrl &VM_ENTRY_LOAD_IA32_PAT ) { -if (!kvm_pat_valid (data)) -return 1 ;vmcs_write64 (GUEST_IA32_PAT , data); vcpu->arch.pat = data;break ;
Signature · 1 line
2.21.0
PATCH 3/15 KVM: nVMX: Always sync GUEST_BNDCFGS when it comes from vmcs01
In reply to Sean ChristophersonIf L1 does not set VM_ENTRY_LOAD_BNDCFGS, then L1's BNDCFGS value must be propagated to vmcs02 since KVM always runs with VM_ENTRY_LOAD_BNDCFGS when MPX is supported. Because the value effectively comes from vmcs01, vmcs02 must be updated even if vmcs12 is clean.
--- arch/x86/kvm/vmx/nested.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/nested.c +6 -7
@@ -2231,13 +2231,9 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)set_cr4_guest_host_mask (vmx); -if (kvm_mpx_supported ()) { -if (vmx->nested.nested_run_pending && - (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )) -vmcs_write64 (GUEST_BNDCFGS , vmcs12->guest_bndcfgs); -else -vmcs_write64 (GUEST_BNDCFGS , vmx->nested.vmcs01_guest_bndcfgs); - } +if (kvm_mpx_supported () && vmx->nested.nested_run_pending && + (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )) +vmcs_write64 (GUEST_BNDCFGS , vmcs12->guest_bndcfgs); }/*
@@ -2280,6 +2276,9 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,kvm_set_dr (vcpu,7 , vcpu->arch.dr7);vmcs_write64 (GUEST_IA32_DEBUGCTL , vmx->nested.vmcs01_debugctl); } +if (kvm_mpx_supported () && (!vmx->nested.nested_run_pending || + !(vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS ))) +vmcs_write64 (GUEST_BNDCFGS , vmx->nested.vmcs01_guest_bndcfgs);vmx_set_rflags (vcpu, vmcs12->guest_rflags);/* EXCEPTION_BITMAP and CR0_GUEST_HOST_MASK should basically be the
Signature · 1 line
2.21.0
PATCH 4/15 KVM: nVMX: Write ENCLS-exiting bitmap once per vmcs02
In reply to Sean ChristophersonKVM doesn't yet support SGX virtualization, i.e. writes a constant value to ENCLS_EXITING_BITMAP so that it can intercept ENCLS and inject a #UD.
--- arch/x86/kvm/vmx/nested.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/nested.c +3 -3
@@ -1948,6 +1948,9 @@ static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx)if (enable_pml)vmcs_write64 (PML_ADDRESS ,page_to_phys (vmx->pml_pg)); +if (cpu_has_vmx_encls_vmexit ()) +vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); +/* * Set the MSR load/store lists to match L0's settings. Only the * addresses are constant (for vmcs02), the counts can change based
@@ -2070,9 +2073,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)if (exec_control &SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES )vmcs_write64 (APIC_ACCESS_ADDR ,-1ull ); -if (exec_control &SECONDARY_EXEC_ENCLS_EXITING ) -vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull ); -vmcs_write32 (SECONDARY_VM_EXEC_CONTROL , exec_control); }
Signature · 1 line
2.21.0
PATCH 5/15 KVM: nVMX: Don't rewrite GUEST_PML_INDEX during nested VM-Entry
In reply to Sean ChristophersonEmulation of GUEST_PML_INDEX for a nested VMM is a bit weird. Because L0 flushes the PML on every VM-Exit, the value in vmcs02 at the time of VM-Enter is a constant -1, regardless of what L1 thinks/wants.
--- arch/x86/kvm/vmx/nested.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)
arch/x86/kvm/vmx/nested.c +9 -11
@@ -1945,8 +1945,16 @@ static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx)if (cpu_has_vmx_msr_bitmap ())vmcs_write64 (MSR_BITMAP ,__pa (vmx->nested.vmcs02.msr_bitmap)); -if (enable_pml) +/* +* Conceptually we want to copy the PML address and index from vmcs01 +* here, and then back to vmcs01 on nested vmexit. But since we always +* flush the log on each vmexit and never change the PML address (once +* set), both fields are effectively constant in vmcs02. +*/ +if (enable_pml) {vmcs_write64 (PML_ADDRESS ,page_to_phys (vmx->pml_pg)); +vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); + }if (cpu_has_vmx_encls_vmexit ())vmcs_write64 (ENCLS_EXITING_BITMAP ,-1ull );
@@ -2106,16 +2114,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) exec_control |=VM_EXIT_LOAD_IA32_EFER ;vm_exit_controls_init (vmx, exec_control); -/* -* Conceptually we want to copy the PML address and index from -* vmcs01 here, and then back to vmcs01 on nested vmexit. But, -* since we always flush the log on each vmexit and never change -* the PML address (once set), this happens to be equivalent to -* simply resetting the index in vmcs02. -*/ -if (enable_pml) -vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); -/* * Interrupt/Exception Fields */
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 05/15] KVM: nVMX: Don't rewrite GUEST_PML_INDEX during nested VM-Entry
arch/x86/kvm/vmx/nested.c static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx) · line 1957
Quoted 11 lines · Sean Christopherson
arch/x86/kvm/vmx/nested.c +9
-1-if (enable_pml) +/* +* Conceptually we want to copy the PML address and index from vmcs01 +* here, and then back to vmcs01 on nested vmexit. But since we always +* flush the log on each vmexit and never change the PML address (once +* set), both fields are effectively constant in vmcs02. +*/ +if (enable_pml) {vmcs_write64 (PML_ADDRESS ,page_to_phys (vmx->pml_pg)); +vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); + }
Yeah, it will be rewritten in vmx_flush_pml_buffer.
Just a little rephrasing of the comment:
+ * The PML address never changes, so it is constant in vmcs02.
+ * Conceptually we want to copy the PML index from vmcs01 here,
+ * and then back to vmcs01 on nested vmexit. But since we flush
+ * the log and reset GUEST_PML_INDEX on each vmexit, the PML
+ * index is also effectively constant in vmcs02.
Paolo
PATCH 6/15 KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
In reply to Sean ChristophersonWhen switching between vmcs01 and vmcs02, KVM isn't actually switching between guest and host. If guest state is already loaded (the likely, if not guaranteed, case), keep the guest state loaded and manually swap the loaded_cpu_state pointer after propagating saved host state to the new vmcs0{1,2}.
Avoiding the switch between guest and host reduces the latency of switching between vmcs01 and vmcs02 by several hundred cycles, and reduces the roundtrip time of a nested VM by upwards of 1000 cycles.
--- arch/x86/kvm/vmx/nested.c | 18 +++++++++++++- arch/x86/kvm/vmx/vmx.c | 52 ++++++++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 48 insertions(+), 25 deletions(-)
arch/x86/kvm/vmx/nested.c +17 -1
@@ -241,15 +241,31 @@ static void free_nested(struct kvm_vcpu *vcpu)static void vmx_switch_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vmcs_host_state *src; +struct loaded_vmcs *prev;int cpu;if (vmx->loaded_vmcs == vmcs)return ; cpu =get_cpu (); -vmx_vcpu_put (vcpu); + prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs;vmx_vcpu_load (vcpu, cpu); + +if (likely (prev)) { + src = &prev->host_state; + +vmx_set_host_fs_gs (&vmcs->host_state, src->fs_sel, src->gs_sel, + src->fs_base, src->gs_base); + + vmcs->host_state.ldt_sel = src->ldt_sel; +#ifdef CONFIG_X86_64 + vmcs->host_state.ds_sel = src->ds_sel; + vmcs->host_state.es_sel = src->es_sel; +#endif + vmx->loaded_cpu_state = vmcs; + }put_cpu ();vm_entry_controls_reset_shadow (vmx);
arch/x86/kvm/vmx/vmx.c +29 -23
70 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +2 -1
@@ -293,11 +293,12 @@ struct kvm_vmx {bool nested_vmx_allowed (struct kvm_vcpu *vcpu);void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu); -void vmx_vcpu_put (struct kvm_vcpu *vcpu);int allocate_vpid (void );void free_vpid (int vpid);void vmx_set_constant_host_state (struct vcpu_vmx *vmx);void vmx_prepare_switch_to_guest (struct kvm_vcpu *vcpu); +void vmx_set_host_fs_gs (struct vmcs_host_state *host,u16 fs_sel,u16 gs_sel, +unsigned long fs_base,unsigned long gs_base);int vmx_get_cpl (struct kvm_vcpu *vcpu);unsigned long vmx_get_rflags (struct kvm_vcpu *vcpu);void vmx_set_rflags (struct kvm_vcpu *vcpu,unsigned long rflags);
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 06/15] KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
Quoted 42 lines · Sean Christopherson
When switching between vmcs01 and vmcs02, KVM isn't actually switching between guest and host. If guest state is already loaded (the likely, if not guaranteed, case), keep the guest state loaded and manually swap the
loaded_cpu_statepointer after propagating saved host state to the new vmcs0{1,2}.Avoiding the switch between guest and host reduces the latency of switching between vmcs01 and vmcs02 by several hundred cycles, and reduces the roundtrip time of a nested VM by upwards of 1000 cycles.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 18 +++++++++++++- arch/x86/kvm/vmx/vmx.c | 52 ++++++++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 48 insertions(+), 25 deletions(-)arch/x86/kvm/vmx/nested.c +8
-1@@ -241,15 +241,31 @@ static void free_nested(struct kvm_vcpu *vcpu)static void vmx_switch_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vmcs_host_state *src; +struct loaded_vmcs *prev;int cpu;if (vmx->loaded_vmcs == vmcs)return ; cpu =get_cpu (); -vmx_vcpu_put (vcpu); + prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs;vmx_vcpu_load (vcpu, cpu); + +if (likely (prev)) { + src = &prev->host_state; + + vmx_set_host_fs_gs(&vmcs->host_state, src->fs_sel, src->gs_sel,
arch/x86/kvm/vmx/nested.c static void free_nested(struct kvm_vcpu *vcpu) · line 269
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +9
-0+ src->fs_base, src->gs_base); + + vmcs->host_state.ldt_sel = src->ldt_sel; +#ifdef CONFIG_X86_64 + vmcs->host_state.ds_sel = src->ds_sel; + vmcs->host_state.es_sel = src->es_sel; +#endif + vmx->loaded_cpu_state = vmcs; + }put_cpu ();
I'd like to extract this into a separate function:
arch/x86/kvm/vmx/nested.c +23 -17
@@ -248,34 +248,40 @@ static void free_nested(struct kvm_vcpu *vcpu)free_loaded_vmcs (&vmx->nested.vmcs02); } +static void vmx_sync_vmcs_host_state (struct vcpu_vmx *vmx) +{ +struct loaded_vmcs *prev = vmx->loaded_cpu_state; +struct loaded_vmcs *cur; +struct vmcs_host_state *dest, *src; + +if (unlikely (!prev)) +return ; + + cur = &vmx->loaded_vmcs; + src = &prev->host_state; + dest = &cur->host_state; + +vmx_set_host_fs_gs (dest, src->fs_sel, src->gs_sel, src->fs_base, src->gs_base); + dest->ldt_sel = src->ldt_sel; +#ifdef CONFIG_X86_64 + dest->ds_sel = src->ds_sel; + dest->es_sel = src->es_sel; +#endif + vmx->loaded_cpu_state = cur; +} +static void vmx_switch_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {struct vcpu_vmx *vmx =to_vmx (vcpu); -struct vmcs_host_state *src; -struct loaded_vmcs *prev;int cpu;if (vmx->loaded_vmcs == vmcs)return ; cpu =get_cpu (); - prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs;vmx_vcpu_load (vcpu, cpu); - -if (likely (prev)) { - src = &prev->host_state; - -vmx_set_host_fs_gs (&vmcs->host_state, src->fs_sel, src->gs_sel, - src->fs_base, src->gs_base); - - vmcs->host_state.ldt_sel = src->ldt_sel; -#ifdef CONFIG_X86_64 - vmcs->host_state.ds_sel = src->ds_sel; - vmcs->host_state.es_sel = src->es_sel; -#endif - vmx->loaded_cpu_state = vmcs; - } +vmx_sync_vmcs_host_state (vmx);put_cpu ();vm_entry_controls_reset_shadow (vmx);
Paolo
Re: [PATCH 06/15] KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
Quoted 61 lines · Paolo Bonzini
Quoted 42 lines · Sean Christopherson
When switching between vmcs01 and vmcs02, KVM isn't actually switching between guest and host. If guest state is already loaded (the likely, if not guaranteed, case), keep the guest state loaded and manually swap the
loaded_cpu_statepointer after propagating saved host state to the new vmcs0{1,2}.Avoiding the switch between guest and host reduces the latency of switching between vmcs01 and vmcs02 by several hundred cycles, and reduces the roundtrip time of a nested VM by upwards of 1000 cycles.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 18 +++++++++++++- arch/x86/kvm/vmx/vmx.c | 52 ++++++++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 48 insertions(+), 25 deletions(-)arch/x86/kvm/vmx/nested.c +8
-1@@ -241,15 +241,31 @@ static void free_nested(struct kvm_vcpu *vcpu)static void vmx_switch_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vmcs_host_state *src; +struct loaded_vmcs *prev;int cpu;if (vmx->loaded_vmcs == vmcs)return ; cpu =get_cpu (); -vmx_vcpu_put (vcpu); + prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs;vmx_vcpu_load (vcpu, cpu); + +if (likely (prev)) { + src = &prev->host_state; + + vmx_set_host_fs_gs(&vmcs->host_state, src->fs_sel, src->gs_sel,arch/x86/kvm/vmx/nested.c static void free_nested(struct kvm_vcpu *vcpu) · line 269
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +9
-0+ src->fs_base, src->gs_base); + + vmcs->host_state.ldt_sel = src->ldt_sel; +#ifdef CONFIG_X86_64 + vmcs->host_state.ds_sel = src->ds_sel; + vmcs->host_state.es_sel = src->es_sel; +#endif + vmx->loaded_cpu_state = vmcs; + }put_cpu ();I'd like to extract this into a separate function:
arch/x86/kvm/vmx/nested.c +0
-0
arch/x86/kvm/vmx/nested.c static void free_nested(struct kvm_vcpu *vcpu) · line 251
Quoted 4 lines
arch/x86/kvm/vmx/nested.c +1
-0@@ -248,34 +248,40 @@ static void free_nested(struct kvm_vcpu *vcpu)free_loaded_vmcs (&vmx->nested.vmcs02); } +static void vmx_sync_vmcs_host_state (struct vcpu_vmx *vmx)
What about taking the vmcs pointers, and using old/new instead of prev/cur? Calling it prev is wonky since it's pulled from the current value of loaded_cpu_state, especially since cur is the same type. That oddity is also why I grabbed prev before setting loaded_vmcs, it just felt wrong even though they really are two separate things.
static void vmx_sync_vmcs_host_state(struct vcpu_vmx *vmx,
struct loaded_vmcs *old, struct loaded_vmcs *new)
{
...
}{
vmx_sync_vmcs_host_state(vmx, vmx->loaded_cpu_state, vmcs);
}Quoted 55 lines
arch/x86/kvm/vmx/nested.c +22
-17+{ +struct loaded_vmcs *prev = vmx->loaded_cpu_state; +struct loaded_vmcs *cur; +struct vmcs_host_state *dest, *src; + +if (unlikely (!prev)) +return ; + + cur = &vmx->loaded_vmcs; + src = &prev->host_state; + dest = &cur->host_state; + +vmx_set_host_fs_gs (dest, src->fs_sel, src->gs_sel, src->fs_base, src->gs_base); + dest->ldt_sel = src->ldt_sel; +#ifdef CONFIG_X86_64 + dest->ds_sel = src->ds_sel; + dest->es_sel = src->es_sel; +#endif + vmx->loaded_cpu_state = cur; +} +static void vmx_switch_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {struct vcpu_vmx *vmx =to_vmx (vcpu); -struct vmcs_host_state *src; -struct loaded_vmcs *prev;int cpu;if (vmx->loaded_vmcs == vmcs)return ; cpu =get_cpu (); - prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs;vmx_vcpu_load (vcpu, cpu); - -if (likely (prev)) { - src = &prev->host_state; - -vmx_set_host_fs_gs (&vmcs->host_state, src->fs_sel, src->gs_sel, - src->fs_base, src->gs_base); - - vmcs->host_state.ldt_sel = src->ldt_sel; -#ifdef CONFIG_X86_64 - vmcs->host_state.ds_sel = src->ds_sel; - vmcs->host_state.es_sel = src->es_sel; -#endif - vmx->loaded_cpu_state = vmcs; - } +vmx_sync_vmcs_host_state (vmx);put_cpu ();vm_entry_controls_reset_shadow (vmx);Paolo
Paolo Bonzini
maintainerRe: [PATCH 06/15] KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
Quoted 9 lines · Sean Christopherson
What about taking the vmcs pointers, and using old/new instead of prev/cur? Calling it prev is wonky since it's pulled from the current value of
loaded_cpu_state, especially since cur is the same type. That oddity is also why I grabbed prev before settingloaded_vmcs, it just felt wrong even though they really are two separate things.static void vmx_sync_vmcs_host_state(struct
vcpu_vmx*vmx,
structloaded_vmcs*old, structloaded_vmcs*new)
I had it like that in the beginning actually. But the idea of this function is that because we're switching vmcs's, the host register fields have to be moved to the VMCS that will be used next. I don't see how it would be used with old and new being anything other than vmx->loaded_cpu_state and vmx->loaded_vmcs and, because we're switching VMCS, those are the "previously" active VMCS and the "currently" active VMCS.
What would also make sense, is to change loaded_cpu_state to a bool (it must always be equal to loaded_vmcs anyway) and make the prototype something like this:
static void vmx_sync_vmcs_host_state(struct vcpu_vmx *vmx,
struct loaded_vmcs *prev)
I'll send a patch.
Paolo
Re: [PATCH 06/15] KVM: nVMX: Don't "put" vCPU or host state when switching VMCS
Quoted 23 lines · Paolo Bonzini
Quoted 9 lines · Sean Christopherson
What about taking the vmcs pointers, and using old/new instead of prev/cur? Calling it prev is wonky since it's pulled from the current value of
loaded_cpu_state, especially since cur is the same type. That oddity is also why I grabbed prev before settingloaded_vmcs, it just felt wrong even though they really are two separate things.static void vmx_sync_vmcs_host_state(struct
vcpu_vmx*vmx,
structloaded_vmcs*old, structloaded_vmcs*new)I had it like that in the beginning actually. But the idea of this function is that because we're switching vmcs's, the host register fields have to be moved to the VMCS that will be used next. I don't see how it would be used with old and new being anything other than
vmx->loaded_cpu_stateandvmx->loaded_vmcsand, because we're switching VMCS, those are the "previously" active VMCS and the "currently" active VMCS.What would also make sense, is to change
loaded_cpu_stateto a bool (it must always be equal toloaded_vmcsanyway) and make the prototype something like this:
Quoted 5 lines
static void vmx_sync_vmcs_host_state(struct
vcpu_vmx*vmx,
structloaded_vmcs*prev)I'll send a patch.
Works for me. The only reason I made loaded_cpu_state was so that vmx_prepare_switch_to_host() could WARN on it diverging from loaded_vmcs. Seeing as how that WARN has never fired, I'm comfortable making it a bool.
PATCH 7/15 KVM: nVMX: Don't reread VMCS-agnostic state when switching VMCS
In reply to Sean ChristophersonWhen switching between vmcs01 and vmcs02, there is no need to update state tracking for values that aren't tied to any particular VMCS as the per-vCPU values are already up-to-date (vmx_switch_vmcs() can only be called when the vCPU is loaded).
Avoiding the update eliminates a RDMSR, and potentially a RDPKRU and posted-interrupt updated (cmpxchg64() and more).
--- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 18 +++++++++++++----- arch/x86/kvm/vmx/vmx.h | 2 +- 3 files changed, 15 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -1
@@ -251,7 +251,7 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) cpu =get_cpu (); prev = vmx->loaded_cpu_state; vmx->loaded_vmcs = vmcs; -vmx_vcpu_load (vcpu, cpu); +__vmx_vcpu_load (vcpu, cpu);if (likely (prev)) { src = &prev->host_state;
arch/x86/kvm/vmx/vmx.c +13 -5
@@ -1231,11 +1231,7 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)pi_set_on (pi_desc); } -/* -* Switches to specified vcpu, until a matching vcpu_put(), but assumes -* vcpu mutex is already taken. -*/ -void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +void __vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);bool already_loaded = vmx->loaded_vmcs->cpu == cpu;
@@ -1296,8 +1292,20 @@ void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (kvm_has_tsc_control && vmx->current_tsc_ratio != vcpu->arch.tsc_scaling_ratio)decache_tsc_multiplier (vmx); +} + +/* +* Switches to specified vcpu, until a matching vcpu_put(), but assumes +* vcpu mutex is already taken. +*/ +static void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +__vmx_vcpu_load (vcpu, cpu);vmx_vcpu_pi_load (vcpu, cpu); + vmx->host_pkru =read_pkru (); vmx->host_debugctlmsr =get_debugctlmsr (); }
arch/x86/kvm/vmx/vmx.h +1 -1
@@ -292,7 +292,7 @@ struct kvm_vmx { };bool nested_vmx_allowed (struct kvm_vcpu *vcpu); -void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +void __vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);int allocate_vpid (void );void free_vpid (int vpid);void vmx_set_constant_host_state (struct vcpu_vmx *vmx);
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 07/15] KVM: nVMX: Don't reread VMCS-agnostic state when switching VMCS
arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) · line 1234
Quoted 2 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +1
-1-void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +void __vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu)
Let's call this vmx_vcpu_load_vmcs.
Paolo
PATCH 8/15 KVM: nVMX: Don't speculatively write virtual-APIC page address
In reply to Sean ChristophersonThe VIRTUAL_APIC_PAGE_ADDR in vmcs02 is guaranteed to be updated before it is consumed by hardware, either in nested_vmx_enter_non_root_mode() or via the KVM_REQ_GET_VMCS12_PAGES callback. Avoid an extra VMWRITE and only stuff a bad value into vmcs02 when mapping vmcs12's address fails. This also eliminates the need for extra comments to connect the dots between prepare_vmcs02_early() and nested_get_vmcs12_pages().
--- arch/x86/kvm/vmx/nested.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)
arch/x86/kvm/vmx/nested.c +8 -13
@@ -2038,20 +2038,13 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) exec_control &= ~CPU_BASED_TPR_SHADOW ; exec_control |= vmcs12->cpu_based_vm_exec_control; -/* -* Write an illegal value to VIRTUAL_APIC_PAGE_ADDR. Later, if -* nested_get_vmcs12_pages can't fix it up, the illegal value -* will result in a VM entry failure. -*/ -if (exec_control &CPU_BASED_TPR_SHADOW ) { -vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR ,-1ull ); +if (exec_control &CPU_BASED_TPR_SHADOW )vmcs_write32 (TPR_THRESHOLD , vmcs12->tpr_threshold); - }else {#ifdef CONFIG_X86_64 + else exec_control |= CPU_BASED_CR8_LOAD_EXITING |CPU_BASED_CR8_STORE_EXITING ;#endif - }/* * A vmexit (to either L1 hypervisor or L0 userspace) is always needed
@@ -2869,10 +2862,6 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)if (nested_cpu_has (vmcs12,CPU_BASED_TPR_SHADOW )) { map = &vmx->nested.virtual_apic_map; -/* -* If translation failed, VM entry will fail because -* prepare_vmcs02 set VIRTUAL_APIC_PAGE_ADDR to -1ull. -*/ if (!kvm_vcpu_map (vcpu,gpa_to_gfn (vmcs12->virtual_apic_page_addr), map)) {vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR ,pfn_to_hpa (map->pfn)); } else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) &&
@@ -2888,6 +2877,12 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)*/ vmcs_clear_bits (CPU_BASED_VM_EXEC_CONTROL ,CPU_BASED_TPR_SHADOW ); + }else { +/* +* Write an illegal value to VIRTUAL_APIC_PAGE_ADDR to +* force VM-Entry to fail. +*/ +vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR ,-1ull ); } }
Signature · 1 line
2.21.0
PATCH 9/15 KVM: nVMX: Don't speculatively write APIC-access page address
In reply to Sean ChristophersonIf nested_get_vmcs12_pages() fails to map L1's APIC_ACCESS_ADDR into L2, then it disables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES in vmcs02. In other words, the APIC_ACCESS_ADDR in vmcs02 is guaranteed to be written with the correct value before being consumed by hardware, drop the unneessary VMWRITE.
--- arch/x86/kvm/vmx/nested.c | 8 -------- 1 file changed, 8 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -8
@@ -2082,14 +2082,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_write16 (GUEST_INTR_STATUS , vmcs12->guest_intr_status); -/* -* Write an illegal value to APIC_ACCESS_ADDR. Later, -* nested_get_vmcs12_pages will either fix it up or -* remove the VM execution control. -*/ -if (exec_control &SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ) -vmcs_write64 (APIC_ACCESS_ADDR ,-1ull ); -vmcs_write32 (SECONDARY_VM_EXEC_CONTROL , exec_control); }
Signature · 1 line
2.21.0
PATCH 10/15 KVM: nVMX: Update vmcs12 for MSR_IA32_CR_PAT when it's written
In reply to Sean ChristophersonAs alluded to by the TODO comment, KVM unconditionally intercepts writes to the PAT MSR. In the unlikely event that L1 allows L2 to write L1's PAT directly but saves L2's PAT on VM-Exit, update vmcs12 when L2 writes the PAT. This eliminates the need to VMREAD the value from vmcs02 on VM-Exit as vmcs12 is already up to date in all situations.
--- arch/x86/kvm/vmx/nested.c | 4 ---- arch/x86/kvm/vmx/vmx.c | 4 ++++ 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -4
@@ -3483,10 +3483,6 @@ static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) vmcs12->guest_ia32_debugctl =vmcs_read64 (GUEST_IA32_DEBUGCTL ); } -/* TODO: These cannot have changed unless we have MSR bitmaps and -* the relevant bit asks not to trap the change */ -if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_IA32_PAT ) - vmcs12->guest_ia32_pat =vmcs_read64 (GUEST_IA32_PAT );if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_IA32_EFER ) vmcs12->guest_ia32_efer = vcpu->arch.efer; vmcs12->guest_sysenter_cs =vmcs_read32 (GUEST_SYSENTER_CS );
arch/x86/kvm/vmx/vmx.c +4 -0
@@ -1913,6 +1913,10 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!kvm_pat_valid (data))return 1 ; +if (is_guest_mode (vcpu) && +get_vmcs12 (vcpu)->vm_exit_controls &VM_EXIT_SAVE_IA32_PAT ) +get_vmcs12 (vcpu)->guest_ia32_pat = data; +if (vmcs_config.vmentry_ctrl &VM_ENTRY_LOAD_IA32_PAT ) {vmcs_write64 (GUEST_IA32_PAT , data); vcpu->arch.pat = data;
Signature · 1 line
2.21.0
PATCH 11/15 KVM: nVMX: Update vmcs12 for SYSENTER MSRs when they're written
In reply to Sean ChristophersonFor L2, KVM always intercepts WRMSR to SYSENTER MSRs. Update vmcs12 in the WRMSR handler so that they don't need to be (re)read from vmcs02 on every nested VM-Exit.
--- arch/x86/kvm/vmx/nested.c | 3 --- arch/x86/kvm/vmx/vmx.c | 6 ++++++ 2 files changed, 6 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -3
@@ -3485,9 +3485,6 @@ static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12)if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_IA32_EFER ) vmcs12->guest_ia32_efer = vcpu->arch.efer; - vmcs12->guest_sysenter_cs =vmcs_read32 (GUEST_SYSENTER_CS ); - vmcs12->guest_sysenter_esp =vmcs_readl (GUEST_SYSENTER_ESP ); - vmcs12->guest_sysenter_eip =vmcs_readl (GUEST_SYSENTER_EIP );if (kvm_mpx_supported ()) vmcs12->guest_bndcfgs =vmcs_read64 (GUEST_BNDCFGS ); }
arch/x86/kvm/vmx/vmx.c +6 -0
@@ -1831,12 +1831,18 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)break ;#endif case MSR_IA32_SYSENTER_CS : +if (is_guest_mode (vcpu)) +get_vmcs12 (vcpu)->guest_sysenter_cs = data;vmcs_write32 (GUEST_SYSENTER_CS , data);break ;case MSR_IA32_SYSENTER_EIP : +if (is_guest_mode (vcpu)) +get_vmcs12 (vcpu)->guest_sysenter_eip = data;vmcs_writel (GUEST_SYSENTER_EIP , data);break ;case MSR_IA32_SYSENTER_ESP : +if (is_guest_mode (vcpu)) +get_vmcs12 (vcpu)->guest_sysenter_esp = data;vmcs_writel (GUEST_SYSENTER_ESP , data);break ;case MSR_IA32_POWER_CTL :
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 11/15] KVM: nVMX: Update vmcs12 for SYSENTER MSRs when they're written
arch/x86/kvm/vmx/nested.c static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) · line 3490
Quoted 3 lines · Sean Christopherson
arch/x86/kvm/vmx/nested.c +0
-3- vmcs12->guest_sysenter_cs =vmcs_read32 (GUEST_SYSENTER_CS ); - vmcs12->guest_sysenter_esp =vmcs_readl (GUEST_SYSENTER_ESP ); - vmcs12->guest_sysenter_eip =vmcs_readl (GUEST_SYSENTER_EIP );
I moved these a bit earlier, together with all other fields that are unconditional and simply have to be vmread from the vmcs02 into the vmcs12.
Paolo
Quoted 2 lines
arch/x86/kvm/vmx/nested.c +0
-0if (kvm_mpx_supported ()) vmcs12->guest_bndcfgs =vmcs_read64 (GUEST_BNDCFGS );
PATCH 12/15 KVM: nVMX: Update vmcs12 for MSR_IA32_DEBUGCTLMSR when it's written
In reply to Sean ChristophersonKVM unconditionally intercepts WRMSR to MSR_IA32_DEBUGCTLMSR. In the unlikely event that L1 allows L2 to write L1's MSR_IA32_DEBUGCTLMSR, but but saves L2's value on VM-Exit, update vmcs12 during L2's WRMSR so as to eliminate the need to VMREAD the value from vmcs02 on nested VM-Exit.
--- arch/x86/kvm/vmx/nested.c | 4 +--- arch/x86/kvm/vmx/vmx.c | 8 ++++++++ 2 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -3
@@ -3478,10 +3478,8 @@ static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) (vmcs12->vm_entry_controls & ~VM_ENTRY_IA32E_MODE ) | (vm_entry_controls_get (to_vmx (vcpu)) &VM_ENTRY_IA32E_MODE ); -if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_DEBUG_CONTROLS ) { +if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_DEBUG_CONTROLS )kvm_get_dr (vcpu,7 , (unsigned long *)&vmcs12->guest_dr7); - vmcs12->guest_ia32_debugctl =vmcs_read64 (GUEST_IA32_DEBUGCTL ); - }if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_IA32_EFER ) vmcs12->guest_ia32_efer = vcpu->arch.efer;
arch/x86/kvm/vmx/vmx.c +8 -0
@@ -1848,6 +1848,14 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)case MSR_IA32_POWER_CTL : vmx->msr_ia32_power_ctl = data;break ; +case MSR_IA32_DEBUGCTLMSR : +if (is_guest_mode (vcpu) &&get_vmcs12 (vcpu)->vm_exit_controls & +VM_EXIT_SAVE_DEBUG_CONTROLS ) +get_vmcs12 (vcpu)->guest_ia32_debugctl = data; + + ret =kvm_set_msr_common (vcpu, msr_info); +break ; +case MSR_IA32_BNDCFGS : if (!kvm_mpx_supported() || (!msr_info->host_initiated &&
Signature · 1 line
2.21.0
PATCH 13/15 KVM: nVMX: Update vmcs02 GUEST_IA32_DEBUGCTL only when vmcs12 is dirty
In reply to Sean ChristophersonVMWRITEs to GUEST_IA32_DEBUGCTL from L1 are always intercepted, and unlike GUEST_DR7 there is no funky logic for determining the value.
--- arch/x86/kvm/vmx/nested.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/nested.c +5 -1
@@ -2194,6 +2194,11 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2);vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); } + +if (vmx->nested.nested_run_pending && + (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) +vmcs_write64 (GUEST_IA32_DEBUGCTL , + vmcs12->guest_ia32_debugctl); }if (nested_cpu_has_xsaves (vmcs12))
@@ -2270,7 +2275,6 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) {kvm_set_dr (vcpu,7 , vmcs12->guest_dr7); -vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl); }else {kvm_set_dr (vcpu,7 , vcpu->arch.dr7);vmcs_write64 (GUEST_IA32_DEBUGCTL , vmx->nested.vmcs01_debugctl);
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 13/15] KVM: nVMX: Update vmcs02 GUEST_IA32_DEBUGCTL only when vmcs12 is dirty
Quoted 27 lines · Sean Christopherson
VMWRITEs to
GUEST_IA32_DEBUGCTLfrom L1 are always intercepted, and unlikeGUEST_DR7there is no funky logic for determining the value.Cc Vitaly Kuznetsov <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/nested.c +5
-0@@ -2194,6 +2194,11 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2);vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); } + +if (vmx->nested.nested_run_pending && + (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) +vmcs_write64 (GUEST_IA32_DEBUGCTL , + vmcs12->guest_ia32_debugctl); }if (nested_cpu_has_xsaves (vmcs12))
arch/x86/kvm/vmx/nested.c static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, · line 2280
Quoted 7 lines
arch/x86/kvm/vmx/nested.c +0
-1@@ -2270,7 +2275,6 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) {kvm_set_dr (vcpu,7 , vmcs12->guest_dr7); -vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl); }else {kvm_set_dr (vcpu,7 , vcpu->arch.dr7);vmcs_write64 (GUEST_IA32_DEBUGCTL , vmx->nested.vmcs01_debugctl);
I'm passing on this one. It really gets more complicated and I'm not sure the savings are worth it.
Paolo
PATCH 14/15 KVM: nVMX: Don't update GUEST_BNDCFGS if it's clean in HV eVMCS
In reply to Sean ChristophersonL1 is responsible for dirtying GUEST_GRP1 if it writes GUEST_BNDCFGS.
--- arch/x86/kvm/vmx/nested.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/nested.c +4 -4
@@ -2199,6 +2199,10 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS))vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl); + +if (kvm_mpx_supported () && vmx->nested.nested_run_pending && + (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )) +vmcs_write64 (GUEST_BNDCFGS , vmcs12->guest_bndcfgs); }if (nested_cpu_has_xsaves (vmcs12))
@@ -2234,10 +2238,6 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_write32 (VM_ENTRY_MSR_LOAD_COUNT , vmx->msr_autoload.guest.nr);set_cr4_guest_host_mask (vmx); - -if (kvm_mpx_supported () && vmx->nested.nested_run_pending && - (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )) -vmcs_write64 (GUEST_BNDCFGS , vmcs12->guest_bndcfgs); }/*
Signature · 1 line
2.21.0
PATCH 15/15 KVM: nVMX: Copy PDPTRs to/from vmcs12 only when necessary
In reply to Sean ChristophersonPer Intel's SDM:
... the logical processor uses PAE paging if CR0.PG=1, CR4.PAE=1 and IA32_EFER.LME=0. A VM entry to a guest that uses PAE paging loads the PDPTEs into internal, non-architectural registers based on the setting of the "enable EPT" VM-execution control.
and:
[GUEST_PDPTR] values are saved into the four PDPTE fields as follows:
- If the "enable EPT" VM-execution control is 0 or the logical processor was not using PAE paging at the time of the VM exit, the values saved are undefined.
In other words, if EPT is disabled or the guest isn't using PAE paging, then the PDPTRS aren't consumed by hardware on VM-Entry and are loaded with junk on VM-Exit. From a nesting perspective, all of the above hold true, i.e. KVM can effectively ignore the VMCS PDPTRs. E.g. KVM already loads the PDPTRs from memory when nested EPT is disabled (see nested_vmx_load_cr3()).
Because KVM intercepts setting CR4.PAE, there is no danger of consuming a stale value or crushing L1's VMWRITEs regardless of whether L1 intercepts CR4.PAE. The vmcs12's values are unchanged up until the VM-Exit where L2 sets CR4.PAE, i.e. L0 will see the new PAE state on the subsequent VM-Entry and propagate the PDPTRs from vmcs12 to vmcs02.
--- arch/x86/kvm/vmx/nested.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/nested.c +21 -15
@@ -2184,17 +2184,6 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_writel (GUEST_SYSENTER_ESP , vmcs12->guest_sysenter_esp);vmcs_writel (GUEST_SYSENTER_EIP , vmcs12->guest_sysenter_eip); -/* -* L1 may access the L2's PDPTR, so save them to construct -* vmcs12 -*/ -if (enable_ept) { -vmcs_write64 (GUEST_PDPTR0 , vmcs12->guest_pdptr0); -vmcs_write64 (GUEST_PDPTR1 , vmcs12->guest_pdptr1); -vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2); -vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); - } -if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) vmcs_write64(GUEST_IA32_DEBUGCTL,
@@ -2256,10 +2245,15 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, {struct vcpu_vmx *vmx =to_vmx (vcpu);struct hv_enlightened_vmcs *hv_evmcs = vmx->nested.hv_evmcs; +bool load_guest_pdptrs_vmcs12 = false;if (vmx->nested.dirty_vmcs12 || vmx->nested.hv_evmcs) {prepare_vmcs02_full (vmx, vmcs12); vmx->nested.dirty_vmcs12 = false; + + load_guest_pdptrs_vmcs12 = !vmx->nested.hv_evmcs || + !(vmx->nested.hv_evmcs->hv_clean_fields & +HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 ); }/*
@@ -2366,6 +2360,15 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,return -EINVAL ; } +/* Late preparation of GUEST_PDPTRs now that EFER and CRs are set. */ +if (load_guest_pdptrs_vmcs12 &&nested_cpu_has_ept (vmcs12) && + !is_long_mode (vcpu) &&is_pae (vcpu) &&is_paging (vcpu)) { +vmcs_write64 (GUEST_PDPTR0 , vmcs12->guest_pdptr0); +vmcs_write64 (GUEST_PDPTR1 , vmcs12->guest_pdptr1); +vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2); +vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); + } +/* Shadow page tables on either EPT or shadow page tables. */ if (nested_vmx_load_cr3 (vcpu, vmcs12->guest_cr3,nested_cpu_has_ept (vmcs12), entry_failure_code))
@@ -3467,10 +3470,13 @@ static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12)*/ if (enable_ept) { vmcs12->guest_cr3 =vmcs_readl (GUEST_CR3 ); - vmcs12->guest_pdptr0 =vmcs_read64 (GUEST_PDPTR0 ); - vmcs12->guest_pdptr1 =vmcs_read64 (GUEST_PDPTR1 ); - vmcs12->guest_pdptr2 =vmcs_read64 (GUEST_PDPTR2 ); - vmcs12->guest_pdptr3 =vmcs_read64 (GUEST_PDPTR3 ); +if (nested_cpu_has_ept (vmcs12) && !is_long_mode (vcpu) && +is_pae (vcpu) &&is_paging (vcpu)) { + vmcs12->guest_pdptr0 =vmcs_read64 (GUEST_PDPTR0 ); + vmcs12->guest_pdptr1 =vmcs_read64 (GUEST_PDPTR1 ); + vmcs12->guest_pdptr2 =vmcs_read64 (GUEST_PDPTR2 ); + vmcs12->guest_pdptr3 =vmcs_read64 (GUEST_PDPTR3 ); + } } vmcs12->guest_linear_address =vmcs_readl (GUEST_LINEAR_ADDRESS );
Signature · 1 line
2.21.0
Paolo Bonzini
maintainerRe: [PATCH 15/15] KVM: nVMX: Copy PDPTRs to/from vmcs12 only when necessary
Quoted 73 lines · Sean Christopherson
Per Intel's SDM:
... the logical processor uses PAE paging if CR0.PG=1, CR4.PAE=1 and IA32_EFER.LME=0. A VM entry to a guest that uses PAE paging loads the PDPTEs into internal, non-architectural registers based on the setting of the "enable EPT" VM-execution control.
and:
[
GUEST_PDPTR] values are saved into the four PDPTE fields as follows:- If the "enable EPT" VM-execution control is 0 or the logical processor was not using PAE paging at the time of the VM exit, the values saved are undefined.
In other words, if EPT is disabled or the guest isn't using PAE paging, then the PDPTRS aren't consumed by hardware on VM-Entry and are loaded with junk on VM-Exit. From a nesting perspective, all of the above hold true, i.e. KVM can effectively ignore the VMCS PDPTRs. E.g. KVM already loads the PDPTRs from memory when nested EPT is disabled (see
nested_vmx_load_cr3()).Because KVM intercepts setting CR4.PAE, there is no danger of consuming a stale value or crushing L1's VMWRITEs regardless of whether L1 intercepts CR4.PAE. The vmcs12's values are unchanged up until the VM-Exit where L2 sets CR4.PAE, i.e. L0 will see the new PAE state on the subsequent VM-Entry and propagate the PDPTRs from vmcs12 to vmcs02.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)arch/x86/kvm/vmx/nested.c +5
-11@@ -2184,17 +2184,6 @@ static void prepare_vmcs02_full(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmcs_writel (GUEST_SYSENTER_ESP , vmcs12->guest_sysenter_esp);vmcs_writel (GUEST_SYSENTER_EIP , vmcs12->guest_sysenter_eip); -/* -* L1 may access the L2's PDPTR, so save them to construct -* vmcs12 -*/ -if (enable_ept) { -vmcs_write64 (GUEST_PDPTR0 , vmcs12->guest_pdptr0); -vmcs_write64 (GUEST_PDPTR1 , vmcs12->guest_pdptr1); -vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2); -vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); - } -if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) vmcs_write64(GUEST_IA32_DEBUGCTL,@@ -2256,10 +2245,15 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, {struct vcpu_vmx *vmx =to_vmx (vcpu);struct hv_enlightened_vmcs *hv_evmcs = vmx->nested.hv_evmcs; +bool load_guest_pdptrs_vmcs12 = false;if (vmx->nested.dirty_vmcs12 || vmx->nested.hv_evmcs) {prepare_vmcs02_full (vmx, vmcs12); vmx->nested.dirty_vmcs12 = false; + + load_guest_pdptrs_vmcs12 = !vmx->nested.hv_evmcs || + !(vmx->nested.hv_evmcs->hv_clean_fields & +HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 ); }/* @@ -2366,6 +2360,15 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,return -EINVAL ;
arch/x86/kvm/vmx/nested.c static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, · line 2370
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +8
-0} +/* Late preparation of GUEST_PDPTRs now that EFER and CRs are set. */ +if (load_guest_pdptrs_vmcs12 &&nested_cpu_has_ept (vmcs12) && + !is_long_mode (vcpu) &&is_pae (vcpu) &&is_paging (vcpu)) { +vmcs_write64 (GUEST_PDPTR0 , vmcs12->guest_pdptr0); +vmcs_write64 (GUEST_PDPTR1 , vmcs12->guest_pdptr1); +vmcs_write64 (GUEST_PDPTR2 , vmcs12->guest_pdptr2); +vmcs_write64 (GUEST_PDPTR3 , vmcs12->guest_pdptr3); + }
This probably should be merged into nested_vmx_load_cr3, but something for later. I've just sent a patch to create a new is_pae_paging function that can be used here.
Paolo
Quoted 22 lines
arch/x86/kvm/vmx/nested.c +7
-4/* Shadow page tables on either EPT or shadow page tables. */ if (nested_vmx_load_cr3 (vcpu, vmcs12->guest_cr3,nested_cpu_has_ept (vmcs12), entry_failure_code))@@ -3467,10 +3470,13 @@ static void sync_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12)*/ if (enable_ept) { vmcs12->guest_cr3 =vmcs_readl (GUEST_CR3 ); - vmcs12->guest_pdptr0 =vmcs_read64 (GUEST_PDPTR0 ); - vmcs12->guest_pdptr1 =vmcs_read64 (GUEST_PDPTR1 ); - vmcs12->guest_pdptr2 =vmcs_read64 (GUEST_PDPTR2 ); - vmcs12->guest_pdptr3 =vmcs_read64 (GUEST_PDPTR3 ); +if (nested_cpu_has_ept (vmcs12) && !is_long_mode (vcpu) && +is_pae (vcpu) &&is_paging (vcpu)) { + vmcs12->guest_pdptr0 =vmcs_read64 (GUEST_PDPTR0 ); + vmcs12->guest_pdptr1 =vmcs_read64 (GUEST_PDPTR1 ); + vmcs12->guest_pdptr2 =vmcs_read64 (GUEST_PDPTR2 ); + vmcs12->guest_pdptr3 =vmcs_read64 (GUEST_PDPTR3 ); + } } vmcs12->guest_linear_address =vmcs_readl (GUEST_LINEAR_ADDRESS );
Paolo Bonzini
maintainerQuoted 11 lines · Sean Christopherson
The majority of patches in this series are loosely related optimizations to pick off low(ish) hanging fruit in nested VM-Entry, e.g. there are many VMREADs and VMWRITEs that can be optimized away without too much effort.
The major change (in terms of performance) is to not "put" the vCPU state when switching between vmcs01 and vmcs02, which can reudce the latency of a nested VM-Entry by upwards of 1000 cycles.
A few bug fixes are prepended as they touch code that happens to be modified by the various optimizations.
I've queued the patches locally, but it will be a few days before I can give them adequate testing so I have not yet pushed them to kvm/queue.
Paolo