KVM: VMX: Add caching of EXIT_QUAL and INTR_INFO
cover KVM: VMX: Add caching of EXIT_QUAL and INTR_INFO
Patches 4-5 are the focus of this series, adding caching of vmcs.EXIT_QUALIFICATION and proper caching of vmcs.INTR_INFO (instead of caching it with ad hoc rules about when it's available). Patches 1-3 are prep work to clean up the register caching to ensure correctness when switching between vmcs01 and vmcs02.
The idea for this came about when working on the "unionize exit_reason" series. The nested VM-Exit logic looks at both fields multiple times, which is ok-ish when everything is crammed into one or two functions, but incurs multiple VMREADs when split up. I really didn't want to solve that issue by piling on more cases where vmx->exit_intr_info would be valid, or by duplicating that fragile pattern for exit_qualification.
Paolo, this will conflict with the "unionize exit_reason" series, though the conflict resolution is all mechnical in nature. Let me know if you want me to respin one on top of the other, send a single series, etc...
Sean Christopherson (5):
KVM: nVMX: Invoke ept_save_pdptrs() if and only if PAE paging is
enabled
KVM: nVMX: Reset register cache (available and dirty masks) on VMCS
switch
KVM: nVMX: Drop manual clearing of segment cache on nested VMCS switch
KVM: VMX: Cache vmcs.EXIT_QUALIFICATION using arch avail_reg flags
KVM: VMX: Cache vmcs.EXIT_INTR_INFO using arch avail_reg flags
arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/vmx/nested.c | 29 +++++++------ arch/x86/kvm/vmx/nested.h | 4 +- arch/x86/kvm/vmx/vmx.c | 73 ++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 35 +++++++++++++++- 5 files changed, 86 insertions(+), 57 deletions(-)
Signature · 1 line
2.26.0
PATCH 1/5 KVM: nVMX: Invoke ept_save_pdptrs() if and only if PAE paging is enabled
Invoke ept_save_pdptrs() when restoring L1's host state on a "late" VM-Fail if and only if PAE paging is enabled. This saves a CALL in the common case where L1 is a 64-bit host, and avoids incorrectly marking the PDPTRs as dirty.
WARN if ept_save_pdptrs() is called with PAE disabled now that the nested usage pre-checks is_pae_paging(). Barring a bug in KVM's MMU, attempting to read the PDPTRs with PAE disabled is now impossible.
--- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 13 +++++++------ 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -1
@@ -4250,7 +4250,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)* VMFail, like everything else we just need to ensure our * software model is up-to-date. */ -if (enable_ept) +if (enable_ept &&is_pae_paging (vcpu))ept_save_pdptrs (vcpu);kvm_mmu_reset_context (vcpu);
arch/x86/kvm/vmx/vmx.c +7 -6
@@ -2937,12 +2937,13 @@ void ept_save_pdptrs(struct kvm_vcpu *vcpu) {struct kvm_mmu *mmu = vcpu->arch.walk_mmu; -if (is_pae_paging (vcpu)) { - mmu->pdptrs[0 ] =vmcs_read64 (GUEST_PDPTR0 ); - mmu->pdptrs[1 ] =vmcs_read64 (GUEST_PDPTR1 ); - mmu->pdptrs[2 ] =vmcs_read64 (GUEST_PDPTR2 ); - mmu->pdptrs[3 ] =vmcs_read64 (GUEST_PDPTR3 ); - } +if (WARN_ON_ONCE (!is_pae_paging (vcpu))) +return ; + + mmu->pdptrs[0 ] =vmcs_read64 (GUEST_PDPTR0 ); + mmu->pdptrs[1 ] =vmcs_read64 (GUEST_PDPTR1 ); + mmu->pdptrs[2 ] =vmcs_read64 (GUEST_PDPTR2 ); + mmu->pdptrs[3 ] =vmcs_read64 (GUEST_PDPTR3 );kvm_register_mark_dirty (vcpu,VCPU_EXREG_PDPTR ); }
Signature · 1 line
2.26.0
PATCH 2/5 KVM: nVMX: Reset register cache (available and dirty masks) on VMCS switch
In reply to Sean ChristophersonReset the per-vCPU available and dirty register masks when switching between vmcs01 and vmcs02, as the masks track state relative to the current VMCS. The stale masks don't cause problems in the current code base because the registers are either unconditionally written on nested transitions or, in the case of segment registers, have an additional tracker that is manually reset.
Note, by dropping (previously implicitly, now explicitly) the dirty mask when switching the active VMCS, KVM is technically losing writes to the associated fields. But, the only regs that can be dirtied (RIP, RSP and PDPTRs) are unconditionally written on nested transitions, e.g. explicit writeback is a waste of cycles, and a WARN_ON would be rather pointless.
--- arch/x86/kvm/vmx/nested.c | 1 + arch/x86/kvm/vmx/vmx.c | 7 +------ arch/x86/kvm/vmx/vmx.h | 11 +++++++++++ 3 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -0
@@ -307,6 +307,7 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)vmx_sync_vmcs_host_state (vmx, prev);put_cpu (); +vmx_register_cache_reset (vcpu);vmx_segment_cache_clear (vmx); }
arch/x86/kvm/vmx/vmx.c +1 -6
@@ -6711,12 +6711,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu)loadsegment (es, __USER_DS);#endif - vcpu->arch.regs_avail = ~((1 <<VCPU_REGS_RIP ) | (1 <<VCPU_REGS_RSP ) - | (1 <<VCPU_EXREG_RFLAGS ) - | (1 <<VCPU_EXREG_PDPTR ) - | (1 <<VCPU_EXREG_SEGMENTS ) - | (1 <<VCPU_EXREG_CR3 )); - vcpu->arch.regs_dirty =0 ; +vmx_register_cache_reset (vcpu);pt_guest_exit (vmx);
arch/x86/kvm/vmx/vmx.h +11 -0
@@ -8,6 +8,7 @@#include <asm/intel_pt.h> #include "capabilities.h" +#include "kvm_cache_regs.h" #include "ops.h" #include "vmcs.h"
@@ -447,6 +448,16 @@ static inline void vmx_segment_cache_clear(struct vcpu_vmx *vmx) vmx->segment_cache.bitmask =0 ; } +static inline void vmx_register_cache_reset (struct kvm_vcpu *vcpu) +{ + vcpu->arch.regs_avail = ~((1 <<VCPU_REGS_RIP ) | (1 <<VCPU_REGS_RSP ) + | (1 <<VCPU_EXREG_RFLAGS ) + | (1 <<VCPU_EXREG_PDPTR ) + | (1 <<VCPU_EXREG_SEGMENTS ) + | (1 <<VCPU_EXREG_CR3 )); + vcpu->arch.regs_dirty =0 ; +} +static inline u32 vmx_vmentry_ctrl (void ) {u32 vmentry_ctrl = vmcs_config.vmentry_ctrl;
Signature · 1 line
2.26.0
PATCH 3/5 KVM: nVMX: Drop manual clearing of segment cache on nested VMCS switch
In reply to Sean ChristophersonDrop the call to vmx_segment_cache_clear() in vmx_switch_vmcs() now that the entire register cache is reset when switching the active VMCS, e.g. vmx_segment_cache_test_set() will reset the segment cache due to VCPU_EXREG_SEGMENTS being unavailable.
Move vmx_segment_cache_clear() to vmx.c now that it's no longer invoked by the nested code.
--- arch/x86/kvm/vmx/nested.c | 1 - arch/x86/kvm/vmx/vmx.c | 5 +++++ arch/x86/kvm/vmx/vmx.h | 5 ----- 3 files changed, 5 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -1
@@ -308,7 +308,6 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)put_cpu ();vmx_register_cache_reset (vcpu); -vmx_segment_cache_clear (vmx); }/*
arch/x86/kvm/vmx/vmx.c +5 -0
@@ -437,6 +437,11 @@ static const struct kvm_vmx_segment_field { VMX_SEGMENT_FIELD(LDTR), }; +static inline void vmx_segment_cache_clear (struct vcpu_vmx *vmx) +{ + vmx->segment_cache.bitmask =0 ; +} +static unsigned long host_idt_base;/*
arch/x86/kvm/vmx/vmx.h +0 -5
@@ -443,11 +443,6 @@ BUILD_CONTROLS_SHADOW(pin, PIN_BASED_VM_EXEC_CONTROL) BUILD_CONTROLS_SHADOW(exec, CPU_BASED_VM_EXEC_CONTROL) BUILD_CONTROLS_SHADOW(secondary_exec, SECONDARY_VM_EXEC_CONTROL) -static inline void vmx_segment_cache_clear (struct vcpu_vmx *vmx) -{ - vmx->segment_cache.bitmask =0 ; -} -static inline void vmx_register_cache_reset (struct kvm_vcpu *vcpu) { vcpu->arch.regs_avail = ~((1 << VCPU_REGS_RIP) | (1 << VCPU_REGS_RSP)
Signature · 1 line
2.26.0
PATCH 4/5 KVM: VMX: Cache vmcs.EXIT_QUALIFICATION using arch avail_reg flags
In reply to Sean ChristophersonIntroduce a new "extended register" type, EXIT_INFO_1 (to pair with the nomenclature in .get_exit_info()), and use it to cache VMX's vmcs.EXIT_QUALIFICATION.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/nested.c | 21 ++++++++++----------- arch/x86/kvm/vmx/nested.h | 2 +- arch/x86/kvm/vmx/vmx.c | 28 ++++++++++++++-------------- arch/x86/kvm/vmx/vmx.h | 15 ++++++++++++++- 5 files changed, 40 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -170,6 +170,7 @@ enum kvm_reg {VCPU_EXREG_CR3 ,VCPU_EXREG_RFLAGS ,VCPU_EXREG_SEGMENTS , + VCPU_EXREG_EXIT_INFO_1, }; enum {
arch/x86/kvm/vmx/nested.c +10 -11
@@ -4603,7 +4603,7 @@ static int nested_vmx_get_vmptr(struct kvm_vcpu *vcpu, gpa_t *vmpointer)gva_t gva;struct x86_exception e; -if (get_vmx_mem_address (vcpu,vmcs_readl (EXIT_QUALIFICATION ), +if (get_vmx_mem_address (vcpu,vmx_get_exit_qual (vcpu),vmcs_read32 (VMX_INSTRUCTION_INFO ), false,sizeof (*vmpointer), &gva))return 1 ;
@@ -4868,7 +4868,7 @@ static int handle_vmread(struct kvm_vcpu *vcpu) {struct vmcs12 *vmcs12 =is_guest_mode (vcpu) ?get_shadow_vmcs12 (vcpu) :get_vmcs12 (vcpu); -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);u32 instr_info =vmcs_read32 (VMX_INSTRUCTION_INFO );struct vcpu_vmx *vmx =to_vmx (vcpu);struct x86_exception e;
@@ -4954,7 +4954,7 @@ static int handle_vmwrite(struct kvm_vcpu *vcpu) {struct vmcs12 *vmcs12 =is_guest_mode (vcpu) ?get_shadow_vmcs12 (vcpu) :get_vmcs12 (vcpu); -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);u32 instr_info =vmcs_read32 (VMX_INSTRUCTION_INFO );struct vcpu_vmx *vmx =to_vmx (vcpu);struct x86_exception e;
@@ -5139,7 +5139,7 @@ static int handle_vmptrld(struct kvm_vcpu *vcpu)/* Emulate the VMPTRST instruction */ static int handle_vmptrst (struct kvm_vcpu *vcpu) { -unsigned long exit_qual =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qual =vmx_get_exit_qual (vcpu);u32 instr_info =vmcs_read32 (VMX_INSTRUCTION_INFO );gpa_t current_vmptr =to_vmx (vcpu)->nested.current_vmptr;struct x86_exception e;
@@ -5207,7 +5207,7 @@ static int handle_invept(struct kvm_vcpu *vcpu)/* According to the Intel VMX instruction reference, the memory * operand is read even if it isn't needed (e.g., for type==global) */ -if (get_vmx_mem_address (vcpu,vmcs_readl (EXIT_QUALIFICATION ), +if (get_vmx_mem_address (vcpu,vmx_get_exit_qual (vcpu), vmx_instruction_info, false,sizeof (operand), &gva))return 1 ;if (kvm_read_guest_virt (vcpu, gva, &operand,sizeof (operand), &e)) {
@@ -5289,7 +5289,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)/* according to the intel vmx instruction reference, the memory * operand is read even if it isn't needed (e.g., for type==global) */ -if (get_vmx_mem_address (vcpu,vmcs_readl (EXIT_QUALIFICATION ), +if (get_vmx_mem_address (vcpu,vmx_get_exit_qual (vcpu), vmx_instruction_info, false,sizeof (operand), &gva))return 1 ;if (kvm_read_guest_virt (vcpu, gva, &operand,sizeof (operand), &e)) {
@@ -5419,7 +5419,7 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) fail:nested_vmx_vmexit (vcpu, vmx->exit_reason,vmcs_read32 (VM_EXIT_INTR_INFO ), -vmcs_readl (EXIT_QUALIFICATION )); +vmx_get_exit_qual (vcpu));return 1 ; }
@@ -5470,7 +5470,7 @@ static bool nested_vmx_exit_handled_io(struct kvm_vcpu *vcpu,if (!nested_cpu_has (vmcs12,CPU_BASED_USE_IO_BITMAPS ))return nested_cpu_has (vmcs12,CPU_BASED_UNCOND_IO_EXITING ); - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu); port = exit_qualification >>16 ; size = (exit_qualification &7 ) +1 ;
@@ -5524,7 +5524,7 @@ static bool nested_vmx_exit_handled_msr(struct kvm_vcpu *vcpu,static bool nested_vmx_exit_handled_cr (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12) { -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);int cr = exit_qualification &15 ;int reg;unsigned long val;
@@ -5662,7 +5662,7 @@ bool nested_vmx_exit_reflected(struct kvm_vcpu *vcpu, u32 exit_reason) } trace_kvm_nested_vmexit(kvm_rip_read(vcpu), exit_reason, - vmcs_readl(EXIT_QUALIFICATION), + vmx_get_exit_qual(vcpu), vmx->idt_vectoring_info, intr_info, vmcs_read32(VM_EXIT_INTR_ERROR_CODE),
@@ -5813,7 +5813,6 @@ bool nested_vmx_exit_reflected(struct kvm_vcpu *vcpu, u32 exit_reason) } } -static int vmx_get_nested_state (struct kvm_vcpu *vcpu,struct kvm_nested_state __user *user_kvm_nested_state,u32 user_data_size)
arch/x86/kvm/vmx/nested.h +1 -1
@@ -104,7 +104,7 @@ static inline int nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu, }nested_vmx_vmexit (vcpu, exit_reason, exit_intr_info, -vmcs_readl (EXIT_QUALIFICATION )); +vmx_get_exit_qual (vcpu));return 1 ; }
arch/x86/kvm/vmx/vmx.c +14 -14
@@ -4698,7 +4698,7 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) }if (is_page_fault (intr_info)) { - cr2 =vmcs_readl (EXIT_QUALIFICATION ); + cr2 =vmx_get_exit_qual (vcpu);/* EPT won't cause page fault directly */ WARN_ON_ONCE (!vcpu->arch.apf.host_apf_reason && enable_ept);return kvm_handle_page_fault (vcpu, error_code, cr2,NULL ,0 );
@@ -4714,7 +4714,7 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)kvm_queue_exception_e (vcpu,AC_VECTOR , error_code);return 1 ;case DB_VECTOR : - dr6 =vmcs_readl (EXIT_QUALIFICATION ); + dr6 =vmx_get_exit_qual (vcpu);if (!(vcpu->guest_debug & (KVM_GUESTDBG_SINGLESTEP |KVM_GUESTDBG_USE_HW_BP ))) { vcpu->arch.dr6 &= ~DR_TRAP_BITS ;
@@ -4769,7 +4769,7 @@ static int handle_io(struct kvm_vcpu *vcpu)int size, in, string;unsigned port; - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu); string = (exit_qualification &16 ) !=0 ; ++vcpu->stat.io_exits;
@@ -4860,7 +4860,7 @@ static int handle_cr(struct kvm_vcpu *vcpu)int err;int ret; - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu); cr = exit_qualification &15 ; reg = (exit_qualification >>8 ) &15 ;switch ((exit_qualification >>4 ) &3 ) {
@@ -4937,7 +4937,7 @@ static int handle_dr(struct kvm_vcpu *vcpu)unsigned long exit_qualification;int dr, dr7, reg; - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu); dr = exit_qualification &DEBUG_REG_ACCESS_NUM ;/* First, if DR does not exist, trigger UD */
@@ -5050,7 +5050,7 @@ static int handle_invd(struct kvm_vcpu *vcpu)static int handle_invlpg (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);kvm_mmu_invlpg (vcpu, exit_qualification);return kvm_skip_emulated_instruction (vcpu);
@@ -5082,7 +5082,7 @@ static int handle_xsetbv(struct kvm_vcpu *vcpu)static int handle_apic_access (struct kvm_vcpu *vcpu) {if (likely (fasteoi)) { -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);int access_type, offset; access_type = exit_qualification &APIC_ACCESS_TYPE ;
@@ -5103,7 +5103,7 @@ static int handle_apic_access(struct kvm_vcpu *vcpu)static int handle_apic_eoi_induced (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);int vector = exit_qualification &0xff ;/* EOI-induced VM exit is trap-like and thus no need to adjust IP */
@@ -5113,7 +5113,7 @@ static int handle_apic_eoi_induced(struct kvm_vcpu *vcpu)static int handle_apic_write (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);u32 offset = exit_qualification &0xfff ;/* APIC-write VM exit is trap-like and thus no need to adjust IP */
@@ -5134,7 +5134,7 @@ static int handle_task_switch(struct kvm_vcpu *vcpu) idt_index = (vmx->idt_vectoring_info &VECTORING_INFO_VECTOR_MASK ); type = (vmx->idt_vectoring_info &VECTORING_INFO_TYPE_MASK ); - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu); reason = (u32 )exit_qualification >>30 ;if (reason ==TASK_SWITCH_GATE && idt_v) {
@@ -5184,7 +5184,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)gpa_t gpa;u64 error_code; - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu);/* * EPT violation happened while executing iret from NMI,
@@ -5444,7 +5444,7 @@ static int handle_invpcid(struct kvm_vcpu *vcpu)/* According to the Intel instruction reference, the memory operand * is read even if it isn't needed (e.g., for type==all) */ -if (get_vmx_mem_address (vcpu,vmcs_readl (EXIT_QUALIFICATION ), +if (get_vmx_mem_address (vcpu,vmx_get_exit_qual (vcpu), vmx_instruction_info, false,sizeof (operand), &gva))return 1 ;
@@ -5520,7 +5520,7 @@ static int handle_pml_full(struct kvm_vcpu *vcpu)trace_kvm_pml_full (vcpu->vcpu_id); - exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + exit_qualification =vmx_get_exit_qual (vcpu);/* * PML buffer FULL happened while executing iret from NMI,
@@ -5634,7 +5634,7 @@ static const int kvm_vmx_max_exit_handlers =static void vmx_get_exit_info (struct kvm_vcpu *vcpu,u64 *info1,u64 *info2) { - *info1 =vmcs_readl (EXIT_QUALIFICATION ); + *info1 =vmx_get_exit_qual (vcpu); *info2 =vmcs_read32 (VM_EXIT_INTR_INFO ); }
arch/x86/kvm/vmx/vmx.h +14 -1
@@ -210,6 +210,7 @@ struct vcpu_vmx {*/ bool guest_state_loaded; +unsigned long exit_qualification;u32 exit_intr_info;u32 idt_vectoring_info;ulong rflags;
@@ -449,7 +450,8 @@ static inline void vmx_register_cache_reset(struct kvm_vcpu *vcpu) | (1 << VCPU_EXREG_RFLAGS) | (1 <<VCPU_EXREG_PDPTR ) | (1 <<VCPU_EXREG_SEGMENTS ) - | (1 << VCPU_EXREG_CR3)); + | (1 <<VCPU_EXREG_CR3 ) + | (1 << VCPU_EXREG_EXIT_INFO_1)); vcpu->arch.regs_dirty =0 ; }
@@ -493,6 +495,17 @@ static inline struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu)return &(to_vmx (vcpu)->pi_desc); } +static inline unsigned long vmx_get_exit_qual (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +if (!kvm_register_is_available (vcpu,VCPU_EXREG_EXIT_INFO_1 )) { +kvm_register_mark_available (vcpu,VCPU_EXREG_EXIT_INFO_1 ); + vmx->exit_qualification =vmcs_readl (EXIT_QUALIFICATION ); + } +return vmx->exit_qualification; +} +struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);int alloc_loaded_vmcs (struct loaded_vmcs *loaded_vmcs);
Signature · 1 line
2.26.0
PATCH 5/5 KVM: VMX: Cache vmcs.EXIT_INTR_INFO using arch avail_reg flags
In reply to Sean ChristophersonIntroduce a new "extended register" type, EXIT_INFO_2 (to pair with the nomenclature in .get_exit_info()), and use it to cache VMX's vmcs.EXIT_INTR_INFO. Drop a comment in vmx_recover_nmi_blocking() that is obsoleted by the generic caching mechanism.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/nested.h | 2 +- arch/x86/kvm/vmx/vmx.c | 20 ++++++++------------ arch/x86/kvm/vmx/vmx.h | 14 +++++++++++++- 5 files changed, 25 insertions(+), 16 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -171,6 +171,7 @@ enum kvm_reg {VCPU_EXREG_RFLAGS ,VCPU_EXREG_SEGMENTS ,VCPU_EXREG_EXIT_INFO_1 , + VCPU_EXREG_EXIT_INFO_2, }; enum {
arch/x86/kvm/vmx/nested.c +2 -2
@@ -5418,7 +5418,7 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) fail:nested_vmx_vmexit (vcpu, vmx->exit_reason, -vmcs_read32 (VM_EXIT_INTR_INFO ), +vmx_get_intr_info (vcpu),vmx_get_exit_qual (vcpu));return 1 ; }
@@ -5648,7 +5648,7 @@ static bool nested_vmx_exit_handled_mtf(struct vmcs12 *vmcs12)*/ bool nested_vmx_exit_reflected (struct kvm_vcpu *vcpu,u32 exit_reason) { -u32 intr_info =vmcs_read32 (VM_EXIT_INTR_INFO ); +u32 intr_info =vmx_get_intr_info (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu);struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);
arch/x86/kvm/vmx/nested.h +1 -1
@@ -86,7 +86,7 @@ static inline bool nested_ept_ad_enabled(struct kvm_vcpu *vcpu)static inline int nested_vmx_reflect_vmexit (struct kvm_vcpu *vcpu,u32 exit_reason) { -u32 exit_intr_info =vmcs_read32 (VM_EXIT_INTR_INFO ); +u32 exit_intr_info =vmx_get_intr_info (vcpu);/* * At this point, the exit interruption info in exit_intr_info
arch/x86/kvm/vmx/vmx.c +8 -12
@@ -5635,7 +5635,7 @@ static const int kvm_vmx_max_exit_handlers =static void vmx_get_exit_info (struct kvm_vcpu *vcpu,u64 *info1,u64 *info2) { *info1 =vmx_get_exit_qual (vcpu); - *info2 =vmcs_read32 (VM_EXIT_INTR_INFO ); + *info2 =vmx_get_intr_info (vcpu); }static void vmx_destroy_pml_buffer (struct vcpu_vmx *vmx)
@@ -6298,16 +6298,16 @@ static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu)static void handle_exception_nmi_irqoff (struct vcpu_vmx *vmx) { - vmx->exit_intr_info =vmcs_read32 (VM_EXIT_INTR_INFO ); +u32 intr_info =vmx_get_intr_info (&vmx->vcpu);/* if exit due to PF check for async PF */ -if (is_page_fault (vmx->exit_intr_info)) { +if (is_page_fault (intr_info)) { vmx->vcpu.arch.apf.host_apf_reason =kvm_read_and_reset_pf_reason ();/* Handle machine checks before interrupts are enabled */ - }else if (is_machine_check (vmx->exit_intr_info)) { + }else if (is_machine_check (intr_info)) {kvm_machine_check ();/* We need to handle NMIs before interrupts are enabled */ - }else if (is_nmi (vmx->exit_intr_info)) { + }else if (is_nmi (intr_info)) {kvm_before_interrupt (&vmx->vcpu); asm("int $2" );kvm_after_interrupt (&vmx->vcpu);
@@ -6322,9 +6322,8 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)unsigned long tmp;#endif gate_desc *desc; -u32 intr_info; +u32 intr_info =vmx_get_intr_info (vcpu); - intr_info =vmcs_read32 (VM_EXIT_INTR_INFO );if (WARN_ONCE (!is_external_intr (intr_info),"KVM: unexpected VM-Exit interrupt info: 0x%x" , intr_info))return ;
@@ -6405,11 +6404,8 @@ static void vmx_recover_nmi_blocking(struct vcpu_vmx *vmx)if (enable_vnmi) {if (vmx->loaded_vmcs->nmi_known_unmasked)return ; -/* -* Can't use vmx->exit_intr_info since we're not sure what -* the exit reason is. -*/ - exit_intr_info =vmcs_read32 (VM_EXIT_INTR_INFO ); + + exit_intr_info =vmx_get_intr_info (&vmx->vcpu); unblock_nmi = (exit_intr_info &INTR_INFO_UNBLOCK_NMI ) !=0 ; vector = exit_intr_info &INTR_INFO_VECTOR_MASK ;/*
arch/x86/kvm/vmx/vmx.h +13 -1
@@ -451,7 +451,8 @@ static inline void vmx_register_cache_reset(struct kvm_vcpu *vcpu) | (1 << VCPU_EXREG_PDPTR) | (1 <<VCPU_EXREG_SEGMENTS ) | (1 <<VCPU_EXREG_CR3 ) - | (1 << VCPU_EXREG_EXIT_INFO_1)); + | (1 <<VCPU_EXREG_EXIT_INFO_1 ) + | (1 << VCPU_EXREG_EXIT_INFO_2)); vcpu->arch.regs_dirty =0 ; }
@@ -506,6 +507,17 @@ static inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu)return vmx->exit_qualification; } +static inline u32 vmx_get_intr_info (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +if (!kvm_register_is_available (vcpu,VCPU_EXREG_EXIT_INFO_2 )) { +kvm_register_mark_available (vcpu,VCPU_EXREG_EXIT_INFO_2 ); + vmx->exit_intr_info =vmcs_read32 (VM_EXIT_INTR_INFO ); + } +return vmx->exit_intr_info; +} +struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);int alloc_loaded_vmcs (struct loaded_vmcs *loaded_vmcs);
Signature · 1 line
2.26.0
Paolo Bonzini
maintainerQuoted 13 lines · Sean Christopherson
Patches 4-5 are the focus of this series, adding caching of vmcs.EXIT_QUALIFICATION and proper caching of vmcs.INTR_INFO (instead of caching it with ad hoc rules about when it's available). Patches 1-3 are prep work to clean up the register caching to ensure correctness when switching between vmcs01 and vmcs02.
The idea for this came about when working on the "unionize
exit_reason" series. The nested VM-Exit logic looks at both fields multiple times, which is ok-ish when everything is crammed into one or two functions, but incurs multiple VMREADs when split up. I really didn't want to solve that issue by piling on more cases wherevmx->exit_intr_infowould be valid, or by duplicating that fragile pattern forexit_qualification.
Quoted 3 lines
Paolo, this will conflict with the "unionize
exit_reason" series, though the conflict resolution is all mechnical in nature. Let me know if you want me to respin one on top of the other, send a single series, etc...
Queued, I think I fixed the conflicts right. We'll see if anything explodes when I test it. :)
Paolo
Quoted 17 lines
Sean Christopherson (5):
KVM: nVMX: Invokeept_save_pdptrs()if and only if PAE paging is
enabled
KVM: nVMX: Reset register cache (available and dirty masks) on VMCS
switch
KVM: nVMX: Drop manual clearing of segment cache on nested VMCS switch
KVM: VMX: Cache vmcs.EXIT_QUALIFICATION using archavail_regflags
KVM: VMX: Cache vmcs.EXIT_INTR_INFO using archavail_regflagsarch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/vmx/nested.c | 29 +++++++------ arch/x86/kvm/vmx/nested.h | 4 +- arch/x86/kvm/vmx/vmx.c | 73 ++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 35 +++++++++++++++- 5 files changed, 86 insertions(+), 57 deletions(-)