KVM: x86: Clean up VMX's TLB flushing code
cover KVM: x86: Clean up VMX's TLB flushing code
This series is technically x86 wide, but it only superficially affects SVM, the motivation and primary touchpoints are all about VMX.
The goal of this series to ultimately clean up __vmx_flush_tlb(), which, for me, manages to be extremely confusing despite being only ten lines of code.
The most confusing aspect of __vmx_flush_tlb() is that it is overloaded for multiple uses:
1) TLB flushes in response to a change in KVM's MMU
2) TLB flushes during nested VM-Enter/VM-Exit when VPID is enabled
3) Guest-scoped TLB flushes for paravirt TLB flushing
Handling (2) and (3) in the same flow as (1) is kludgy, because the rules for (1) are quite different than the rules for (2) and (3). They're all squeezed into __vmx_flush_tlb() via the @invalidate_gpa param, which means "invalidate gpa mappings", not "invalidate a specific gpa"; it took me forever and a day to realize that.
To clean things up, handle (2) by directly calling vpid_sync_context() instead of bouncing through __vmx_flush_tlb(), and handle (3) via a dedicated kvm_x86_ops hook. This allows for a less tricky implementation of vmx_flush_tlb() for (1), and (hopefully) clarifies the rules for what mappings must be invalidated when.
Sean Christopherson (10):
KVM: VMX: Use vpid_sync_context() directly when possible
KVM: VMX: Move vpid_sync_vcpu_addr() down a few lines
KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()
KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into
vpid_sync_context()
KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with address
KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
KVM: VMX: Clean up vmx_flush_tlb_gva()
KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()
KVM: VMX: Drop @invalidate_gpa from __vmx_flush_tlb()
KVM: VMX: Fold __vmx_flush_tlb() into vmx_flush_tlb()
arch/x86/include/asm/kvm_host.h | 8 +++++++- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm.c | 14 ++++++++++---- arch/x86/kvm/vmx/nested.c | 12 ++++-------- arch/x86/kvm/vmx/ops.h | 32 +++++++++----------------------- arch/x86/kvm/vmx/vmx.c | 26 +++++++++++++++++--------- arch/x86/kvm/vmx/vmx.h | 19 ++++++++++--------- arch/x86/kvm/x86.c | 8 ++++---- 8 files changed, 62 insertions(+), 59 deletions(-)
Signature · 1 line
2.24.1
PATCH 1/10 KVM: VMX: Use vpid_sync_context() directly when possible
Use vpid_sync_context() directly for flows that run if and only if enable_vpid=1, or more specifically, nested VMX flows that are gated by vmx->nested.msrs.secondary_ctls_high.SECONDARY_EXEC_ENABLE_VPID being set, which is allowed if and only if enable_vpid=1. Because these flows call __vmx_flush_tlb() with @invalidate_gpa=false, the if-statement that decides between INVEPT and INVVPID will always go down the INVVPID path, i.e. call vpid_sync_context() because
"enable_ept && (invalidate_gpa || !enable_vpid)" always evaluates false.
This helps pave the way toward removing @invalidate_gpa and @vpid from __vmx_flush_tlb() and its callers.
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/nested.c +4 -4
@@ -2466,7 +2466,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (nested_cpu_has_vpid (vmcs12) &&nested_has_guest_tlb_tag (vcpu)) {if (vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; -__vmx_flush_tlb (vcpu,nested_get_vpid02 (vcpu), false); +vpid_sync_context (nested_get_vpid02 (vcpu)); } }else {/*
@@ -5154,17 +5154,17 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid02, operand.gla); }else -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;case VMX_VPID_EXTENT_SINGLE_CONTEXT :case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL :if (!operand.vpid)return nested_vmx_failValid (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;case VMX_VPID_EXTENT_ALL_CONTEXT : -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;default :WARN_ON_ONCE (1 );
Signature · 1 line
2.24.1
Re: [PATCH 01/10] KVM: VMX: Use vpid_sync_context() directly when possible
Quoted 33 lines · Sean Christopherson
Use
vpid_sync_context()directly for flows that run if and only if enable_vpid=1, or more specifically, nested VMX flows that are gated byvmx->nested.msrs.secondary_ctls_high.SECONDARY_EXEC_ENABLE_VPIDbeing set, which is allowed if and only if enable_vpid=1. Because these flows call__vmx_flush_tlb()with @invalidate_gpa=false, the if-statement that decides between INVEPT and INVVPID will always go down the INVVPID path, i.e. callvpid_sync_context()because
"enable_ept&& (invalidate_gpa|| !enable_vpid)" always evaluates false.This helps pave the way toward removing @invalidate_gpa and @vpid from
__vmx_flush_tlb()and its callers.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/nested.c +1
-1@@ -2466,7 +2466,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (nested_cpu_has_vpid (vmcs12) &&nested_has_guest_tlb_tag (vcpu)) {if (vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; -__vmx_flush_tlb (vcpu,nested_get_vpid02 (vcpu), false); +vpid_sync_context (nested_get_vpid02 (vcpu)); } }else {/*
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5157
Quoted 5 lines
arch/x86/kvm/vmx/nested.c +1
-1@@ -5154,17 +5154,17 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid02, operand.gla); }else -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);
This is a pre-existing condition but coding style requires braces even for single statements when they were used in another branch.
Quoted 5 lines
arch/x86/kvm/vmx/nested.c +0
-0break ;case VMX_VPID_EXTENT_SINGLE_CONTEXT :case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL :if (!operand.vpid) return nested_vmx_failValid(vcpu,
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5170
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +2
-2VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;case VMX_VPID_EXTENT_ALL_CONTEXT : -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;default :WARN_ON_ONCE (1 );
Seems to be no change indeed,
Signature · 1 line
Vitaly
Re: [PATCH 01/10] KVM: VMX: Use vpid_sync_context() directly when possible
Quoted 35 lines · Vitaly Kuznetsov
Quoted 33 lines · Sean Christopherson
Use
vpid_sync_context()directly for flows that run if and only if enable_vpid=1, or more specifically, nested VMX flows that are gated byvmx->nested.msrs.secondary_ctls_high.SECONDARY_EXEC_ENABLE_VPIDbeing set, which is allowed if and only if enable_vpid=1. Because these flows call__vmx_flush_tlb()with @invalidate_gpa=false, the if-statement that decides between INVEPT and INVVPID will always go down the INVVPID path, i.e. callvpid_sync_context()because
"enable_ept&& (invalidate_gpa|| !enable_vpid)" always evaluates false.This helps pave the way toward removing @invalidate_gpa and @vpid from
__vmx_flush_tlb()and its callers.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/nested.c +1
-1@@ -2466,7 +2466,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (nested_cpu_has_vpid (vmcs12) &&nested_has_guest_tlb_tag (vcpu)) {if (vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; -__vmx_flush_tlb (vcpu,nested_get_vpid02 (vcpu), false); +vpid_sync_context (nested_get_vpid02 (vcpu)); } }else {/* @@ -5154,17 +5154,17 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)
Quoted 8 lines
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5157
Quoted 5 lines
arch/x86/kvm/vmx/nested.c +1
-1__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid02, operand.gla); }else -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);This is a pre-existing condition but coding style requires braces even for single statements when they were used in another branch.
I'll fix this in v2.
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +1
-1break ;case VMX_VPID_EXTENT_SINGLE_CONTEXT :case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL :if (!operand.vpid)return nested_vmx_failValid (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;
Quoted 8 lines
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5170
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +1
-1case VMX_VPID_EXTENT_ALL_CONTEXT : -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);break ;default :WARN_ON_ONCE (1 );Seems to be no change indeed,
Heh, that's about the same level of confidence I had :-)
Quoted 5 lines
Reviewed-by Vitaly Kuznetsov <[email protected]>Signature · 1 line
Vitaly
Paolo Bonzini
maintainerRe: [PATCH 01/10] KVM: VMX: Use vpid_sync_context() directly when possible
Quoted 14 lines · Sean Christopherson
Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +1
-1vmx->nested.last_vpid = vmcs12->virtual_processor_id; -__vmx_flush_tlb (vcpu,nested_get_vpid02 (vcpu), false); +vpid_sync_context (nested_get_vpid02 (vcpu)); } }else {/* Quoted 8 lines
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5157
Quoted 6 lines
arch/x86/kvm/vmx/nested.c +1
-1@@ -5154,17 +5154,17 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid02, operand.gla); }else -__vmx_flush_tlb (vcpu, vpid02, false); +vpid_sync_context (vpid02);This is a pre-existing condition but coding style requires braces even for single statements when they were used in another branch.
Quoted 2 lines
I'll fix this in v2.
Can also remove the braces from the "then" branch.
Paolo
PATCH 2/10 KVM: VMX: Move vpid_sync_vcpu_addr() down a few lines
In reply to Sean ChristophersonMove vpid_sync_vcpu_addr() below vpid_sync_context() so that it can be refactored in a future patch to call vpid_sync_context() directly when the "individual address" INVVPID variant isn't supported.
No functional change intended.
--- arch/x86/kvm/vmx/ops.h | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
arch/x86/kvm/vmx/ops.h +13 -13
@@ -253,19 +253,6 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); } -static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) -{ -if (vpid ==0 ) -return true; - -if (cpu_has_vmx_invvpid_individual_addr ()) { -__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); -return true; - } - -return false; -} -static inline void vpid_sync_vcpu_single (int vpid) {if (vpid ==0 )
@@ -289,6 +276,19 @@ static inline void vpid_sync_context(int vpid)vpid_sync_vcpu_global (); } +static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) +{ +if (vpid ==0 ) +return true; + +if (cpu_has_vmx_invvpid_individual_addr ()) { +__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); +return true; + } + +return false; +} +static inline void ept_sync_global (void ) {__invept (VMX_EPT_EXTENT_GLOBAL ,0 ,0 );
Signature · 1 line
2.24.1
Re: [PATCH 02/10] KVM: VMX: Move vpid_sync_vcpu_addr() down a few lines
Quoted 45 lines · Sean Christopherson
Move
vpid_sync_vcpu_addr()belowvpid_sync_context()so that it can be refactored in a future patch to callvpid_sync_context()directly when the "individual address" INVVPID variant isn't supported.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/ops.h | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)arch/x86/kvm/vmx/ops.h +6
-13@@ -253,19 +253,6 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); } -static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) -{ -if (vpid ==0 ) -return true; - -if (cpu_has_vmx_invvpid_individual_addr ()) { -__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); -return true; - } - -return false; -} -static inline void vpid_sync_vcpu_single (int vpid) {if (vpid ==0 )@@ -289,6 +276,19 @@ static inline void vpid_sync_context(int vpid)vpid_sync_vcpu_global (); } +static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) +{ +if (vpid ==0 ) +return true; + +if (cpu_has_vmx_invvpid_individual_addr ()) {
arch/x86/kvm/vmx/ops.h static inline void vpid_sync_context(int vpid) · line 294
Quoted 10 lines
arch/x86/kvm/vmx/ops.h +7
-0+__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); +return true; + } + +return false; +} +static inline void ept_sync_global (void ) {__invept (VMX_EPT_EXTENT_GLOBAL ,0 ,0 );
Signature · 1 line
Vitaly
PATCH 3/10 KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()
In reply to Sean ChristophersonDirectly invoke vpid_sync_context() to do a global INVVPID when the individual address variant is not supported instead of deferring such behavior to the caller. This allows for additional consolidation of code as the logic is basically identical to the emulation of the individual address variant in handle_invvpid().
No functional change intended.
--- arch/x86/kvm/vmx/ops.h | 12 +++++------- arch/x86/kvm/vmx/vmx.c | 3 +-- 2 files changed, 6 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/ops.h +5 -7
@@ -276,17 +276,15 @@ static inline void vpid_sync_context(int vpid)vpid_sync_vcpu_global (); } -static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) +static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr) {if (vpid ==0 ) -return true; +return ; -if (cpu_has_vmx_invvpid_individual_addr ()) { +if (cpu_has_vmx_invvpid_individual_addr ())__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); -return true; - } - -return false; +else +vpid_sync_context (vpid); }static inline void ept_sync_global (void )
arch/x86/kvm/vmx/vmx.c +1 -2
@@ -2826,8 +2826,7 @@ static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) {int vpid =to_vmx (vcpu)->vpid; -if (!vpid_sync_vcpu_addr (vpid, addr)) -vpid_sync_context (vpid); +vpid_sync_vcpu_addr (vpid, addr);/* * If VPIDs are not supported or enabled, then the above is a no-op.
Signature · 1 line
2.24.1
Re: [PATCH 03/10] KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()
Quoted 46 lines · Sean Christopherson
Directly invoke
vpid_sync_context()to do a global INVVPID when the individual address variant is not supported instead of deferring such behavior to the caller. This allows for additional consolidation of code as the logic is basically identical to the emulation of the individual address variant inhandle_invvpid().No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/ops.h | 12 +++++------- arch/x86/kvm/vmx/vmx.c | 3 +-- 2 files changed, 6 insertions(+), 9 deletions(-)arch/x86/kvm/vmx/ops.h +5
-7@@ -276,17 +276,15 @@ static inline void vpid_sync_context(int vpid)vpid_sync_vcpu_global (); } -static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr) +static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr) {if (vpid ==0 ) -return true; +return ; -if (cpu_has_vmx_invvpid_individual_addr ()) { +if (cpu_has_vmx_invvpid_individual_addr ())__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , vpid, addr); -return true; - } - -return false; +else +vpid_sync_context (vpid); }static inline void ept_sync_global (void )arch/x86/kvm/vmx/vmx.c +0
-0
arch/x86/kvm/vmx/vmx.c static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) · line 2832
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +1
-2@@ -2826,8 +2826,7 @@ static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) {int vpid =to_vmx (vcpu)->vpid; -if (!vpid_sync_vcpu_addr (vpid, addr)) -vpid_sync_context (vpid); +vpid_sync_vcpu_addr (vpid, addr);/* * If VPIDs are not supported or enabled, then the above is a no-op.
Signature · 1 line
Vitaly
PATCH 4/10 KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into vpid_sync_context()
In reply to Sean ChristophersonFold vpid_sync_vcpu_global() and vpid_sync_vcpu_single() into their sole caller. KVM should always prefer the single variant, i.e. the only reason to use the global variant is if the CPU doesn't support invalidating a single VPID, which is the entire purpose of wrapping the calls with vpid_sync_context().
No functional change intended.
--- arch/x86/kvm/vmx/ops.h | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)
arch/x86/kvm/vmx/ops.h +2 -14
@@ -253,29 +253,17 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); } -static inline void vpid_sync_vcpu_single (int vpid) +static inline void vpid_sync_context (int vpid) {if (vpid ==0 )return ;if (cpu_has_vmx_invvpid_single ())__invvpid (VMX_VPID_EXTENT_SINGLE_CONTEXT , vpid,0 ); -} - -static inline void vpid_sync_vcpu_global (void ) -{ -if (cpu_has_vmx_invvpid_global ()) +else __invvpid (VMX_VPID_EXTENT_ALL_CONTEXT ,0 ,0 ); } -static inline void vpid_sync_context (int vpid) -{ -if (cpu_has_vmx_invvpid_single ()) -vpid_sync_vcpu_single (vpid); -else -vpid_sync_vcpu_global (); -} -static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr) {if (vpid ==0 )
Signature · 1 line
2.24.1
Re: [PATCH 04/10] KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into vpid_sync_context()
Quoted 39 lines · Sean Christopherson
Fold
vpid_sync_vcpu_global()andvpid_sync_vcpu_single()into their sole caller. KVM should always prefer the single variant, i.e. the only reason to use the global variant is if the CPU doesn't support invalidating a single VPID, which is the entire purpose of wrapping the calls withvpid_sync_context().No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/ops.h | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)arch/x86/kvm/vmx/ops.h +2
-7@@ -253,29 +253,17 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); } -static inline void vpid_sync_vcpu_single (int vpid) +static inline void vpid_sync_context (int vpid) {if (vpid ==0 )return ;if (cpu_has_vmx_invvpid_single ())__invvpid (VMX_VPID_EXTENT_SINGLE_CONTEXT , vpid,0 ); -} - -static inline void vpid_sync_vcpu_global (void ) -{ -if (cpu_has_vmx_invvpid_global ()) +else __invvpid (VMX_VPID_EXTENT_ALL_CONTEXT ,0 ,0 ); } -static inline void vpid_sync_context (int vpid)
arch/x86/kvm/vmx/ops.h static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa) · line 269
Quoted 10 lines
arch/x86/kvm/vmx/ops.h +0
-7-{ -if (cpu_has_vmx_invvpid_single ()) -vpid_sync_vcpu_single (vpid); -else -vpid_sync_vcpu_global (); -} -static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr) {if (vpid ==0 )
In the original code it's only vpid_sync_vcpu_single() which has 'vpid == 0' check, vpid_sync_vcpu_global() doesn't have it. So in the hypothetical situation when cpu_has_vmx_invvpid_single() is false AND we've e.g. exhausted our VPID space and allocate_vpid() returned zero, the new code just won't do anything while the old one would've done __invvpid(VMX_VPID_EXTENT_ALL_CONTEXT, 0, 0), right?
Signature · 1 line
Vitaly
Re: [PATCH 04/10] KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into vpid_sync_context()
Quoted 50 lines · Vitaly Kuznetsov
Quoted 48 lines · Sean Christopherson
Fold
vpid_sync_vcpu_global()andvpid_sync_vcpu_single()into their sole caller. KVM should always prefer the single variant, i.e. the only reason to use the global variant is if the CPU doesn't support invalidating a single VPID, which is the entire purpose of wrapping the calls withvpid_sync_context().No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/ops.h | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)arch/x86/kvm/vmx/ops.h +2
-14@@ -253,29 +253,17 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); } -static inline void vpid_sync_vcpu_single (int vpid) +static inline void vpid_sync_context (int vpid) {if (vpid ==0 )return ;if (cpu_has_vmx_invvpid_single ())__invvpid (VMX_VPID_EXTENT_SINGLE_CONTEXT , vpid,0 ); -} - -static inline void vpid_sync_vcpu_global (void ) -{ -if (cpu_has_vmx_invvpid_global ()) +else __invvpid (VMX_VPID_EXTENT_ALL_CONTEXT ,0 ,0 ); } -static inline void vpid_sync_context (int vpid) -{ -if (cpu_has_vmx_invvpid_single ()) -vpid_sync_vcpu_single (vpid); -else -vpid_sync_vcpu_global (); -} -static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr) {
Quoted 8 lines
arch/x86/kvm/vmx/ops.h static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa) · line 269
Quoted 1 line
arch/x86/kvm/vmx/ops.h +0
-0if (vpid ==0 )In the original code it's only
vpid_sync_vcpu_single()which has 'vpid == 0' check,vpid_sync_vcpu_global()doesn't have it. So in the hypothetical situation whencpu_has_vmx_invvpid_single()is false AND we've e.g. exhausted our VPID space andallocate_vpid()returned zero, the new code just won't do anything while the old one would've done__invvpid(VMX_VPID_EXTENT_ALL_CONTEXT, 0, 0), right?
Ah rats. I lost track of that functional change between making the commit and writing the changelog.
I'll spin a v2 to rewrite the changelog, and maybe add the "vpid == 0" check in a separate patch.
Paolo Bonzini
maintainerRe: [PATCH 04/10] KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into vpid_sync_context()
Quoted 6 lines · Sean Christopherson
Quoted 6 lines
In the original code it's only
vpid_sync_vcpu_single()which has 'vpid == 0' check,vpid_sync_vcpu_global()doesn't have it. So in the hypothetical situation whencpu_has_vmx_invvpid_single()is false AND we've e.g. exhausted our VPID space andallocate_vpid()returned zero, the new code just won't do anything while the old one would've done__invvpid(VMX_VPID_EXTENT_ALL_CONTEXT, 0, 0), right?
Quoted 6 lines
Ah rats. I lost track of that functional change between making the commit and writing the changelog.
I'll spin a v2 to rewrite the changelog, and maybe add the "vpid == 0" check in a separate patch.
What about this:
arch/x86/kvm/vmx/ops.h +3 -6
@@ -255,13 +255,10 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)static inline void vpid_sync_context (int vpid) { -if (vpid ==0 ) -return ; - -if (cpu_has_vmx_invvpid_single ()) -__invvpid (VMX_VPID_EXTENT_SINGLE_CONTEXT , vpid,0 ); -else +if (!cpu_has_vmx_invvpid_single ())__invvpid (VMX_VPID_EXTENT_ALL_CONTEXT ,0 ,0 ); +else if (vpid !=0 ) +__invvpid (VMX_VPID_EXTENT_SINGLE_CONTEXT , vpid,0 ); }static inline void vpid_sync_vcpu_addr (int vpid,gva_t addr)
PATCH 5/10 KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with address
In reply to Sean ChristophersonUse vpid_sync_vcpu_addr() to emulate the "individual address" variant of INVVPID now that said function handles the fallback case of the (host) CPU not supporting "individual address".
Note, the "vpid == 0" checks in the vpid_sync_*() helpers aren't actually redundant with the "!operand.vpid" check in handle_invvpid(), as the vpid passed to vpid_sync_vcpu_addr() is a KVM (host) controlled value, i.e. vpid02 can be zero even if operand.vpid is non-zero.
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -5
@@ -5150,11 +5150,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) is_noncanonical_address(operand.gla, vcpu))return nested_vmx_failValid (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); -if (cpu_has_vmx_invvpid_individual_addr ()) { -__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , - vpid02, operand.gla); - }else -vpid_sync_context (vpid02); +vpid_sync_vcpu_addr (vpid02, operand.gla);break ;case VMX_VPID_EXTENT_SINGLE_CONTEXT :case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL :
Signature · 1 line
2.24.1
Re: [PATCH 05/10] KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with address
Quoted 23 lines · Sean Christopherson
Use
vpid_sync_vcpu_addr()to emulate the "individual address" variant of INVVPID now that said function handles the fallback case of the (host) CPU not supporting "individual address".Note, the "vpid == 0" checks in the vpid_sync_*() helpers aren't actually redundant with the "!operand.vpid" check in
handle_invvpid(), as the vpid passed tovpid_sync_vcpu_addr()is a KVM (host) controlled value, i.e. vpid02 can be zero even if operand.vpid is non-zero.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/nested.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0@@ -5150,11 +5150,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) is_noncanonical_address(operand.gla, vcpu)) return nested_vmx_failValid(vcpu,
arch/x86/kvm/vmx/nested.c static int handle_invvpid(struct kvm_vcpu *vcpu) · line 5156
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +1
-5VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); -if (cpu_has_vmx_invvpid_individual_addr ()) { -__invvpid (VMX_VPID_EXTENT_INDIVIDUAL_ADDR , - vpid02, operand.gla); - }else -vpid_sync_context (vpid02); +vpid_sync_vcpu_addr (vpid02, operand.gla);break ;case VMX_VPID_EXTENT_SINGLE_CONTEXT :case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL :
Signature · 1 line
Vitaly
PATCH 6/10 KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
In reply to Sean ChristophersonAdd a dedicated hook to handle flushing TLB entries on behalf of the guest, i.e. for a paravirtualized TLB flush, and use it directly instead of bouncing through kvm_vcpu_flush_tlb(). Change the effective VMX implementation to never do INVEPT, i.e. to always flush via INVVPID. The INVEPT performed by __vmx_flush_tlb() when @invalidate_gpa=false and enable_vpid=0 is unnecessary, as it will only flush GPA->HPA mappings; GVA->GPA and GVA->HPA translations are flushed by VM-Enter when VPID is disabled, and changes in the guest pages tables only affect GVA->*PA mappings.
When EPT and VPID are enabled, doing INVVPID is not required (by Intel's architecture) to invalidate GPA mappings, i.e. TLB entries that cache GPA->HPA translations can live across INVVPID as GPA->HPA mappings are associated with an EPTP, not a VPID. The intent of @invalidate_gpa is to inform vmx_flush_tlb() that it needs to "invalidate gpa mappings", i.e. do INVEPT and not simply INVVPID. Other than nested VPID handling, which now calls vpid_sync_context() directly, the only scenario where KVM can safely do INVVPID instead of INVEPT (when EPT is enabled) is if KVM is flushing TLB entries from the guest's perspective, i.e. is invalidating GLA->GPA mappings.
Adding a dedicated ->tlb_flush_guest() paves the way toward removing @invalidate_gpa, which is a potentially dangerous control flag as its meaning is not exactly crystal clear, even for those who are familiar with the subtleties of what mappings Intel CPUs are/aren't allowed to keep across various invalidation scenarios.
--- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/svm.c | 6 ++++++ arch/x86/kvm/vmx/vmx.c | 13 +++++++++++++ arch/x86/kvm/x86.c | 2 +- 4 files changed, 26 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +6 -0
@@ -1108,6 +1108,12 @@ struct kvm_x86_ops {*/ void (*tlb_flush_gva)(struct kvm_vcpu *vcpu,gva_t addr); +/* +* Flush any TLB entries created by the guest. Like tlb_flush_gva(), +* does not need to flush GPA->HPA mappings. +*/ +void (*tlb_flush_guest)(struct kvm_vcpu *vcpu); +void (*run)(struct kvm_vcpu *vcpu);int (*handle_exit)(struct kvm_vcpu *vcpu,enum exit_fastpath_completion exit_fastpath);
arch/x86/kvm/svm.c +6 -0
@@ -5608,6 +5608,11 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)invlpga (gva, svm->vmcb->control.asid); } +static void svm_flush_tlb_guest (struct kvm_vcpu *vcpu) +{ +svm_flush_tlb (vcpu, true); +} +static void svm_prepare_guest_switch (struct kvm_vcpu *vcpu) { }
@@ -7429,6 +7434,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = { .tlb_flush = svm_flush_tlb, .tlb_flush_gva = svm_flush_tlb_gva, + .tlb_flush_guest = svm_flush_tlb_guest, .run = svm_vcpu_run, .handle_exit = handle_exit,
arch/x86/kvm/vmx/vmx.c +13 -0
@@ -2835,6 +2835,18 @@ static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr)*/ } +static void vmx_flush_tlb_guest (struct kvm_vcpu *vcpu) +{ +/* +* vpid_sync_context() is a nop if vmx->vpid==0, e.g. if enable_vpid==0 +* or a vpid couldn't be allocated for this vCPU. VM-Enter and VM-Exit +* are required to flush GVA->{G,H}PA mappings from the TLB if vpid is +* disabled (VM-Enter with vpid enabled and vpid==0 is disallowed), +* i.e. no explicit INVVPID is necessary. +*/ +vpid_sync_context (to_vmx (vcpu)->vpid); +} +static void vmx_decache_cr0_guest_bits (struct kvm_vcpu *vcpu) {ulong cr0_guest_owned_bits = vcpu->arch.cr0_guest_owned_bits;
@@ -7779,6 +7791,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { .tlb_flush = vmx_flush_tlb, .tlb_flush_gva = vmx_flush_tlb_gva, + .tlb_flush_guest = vmx_flush_tlb_guest, .run = vmx_vcpu_run, .handle_exit = vmx_handle_exit,
arch/x86/kvm/x86.c +1 -1
@@ -2675,7 +2675,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)trace_kvm_pv_tlb_flush (vcpu->vcpu_id, st->preempted &KVM_VCPU_FLUSH_TLB );if (xchg (&st->preempted,0 ) &KVM_VCPU_FLUSH_TLB ) -kvm_vcpu_flush_tlb (vcpu, false); + kvm_x86_ops->tlb_flush_guest (vcpu); vcpu->arch.st.preempted =0 ;
Signature · 1 line
2.24.1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Paolo Bonzini
maintainerRe: [PATCH 06/10] KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
Quoted 11 lines · Vitaly Kuznetsov
Quoted 11 lines
arch/x86/kvm/x86.c +1
-1@@ -2675,7 +2675,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)trace_kvm_pv_tlb_flush (vcpu->vcpu_id, st->preempted &KVM_VCPU_FLUSH_TLB );if (xchg (&st->preempted,0 ) &KVM_VCPU_FLUSH_TLB ) -kvm_vcpu_flush_tlb (vcpu, false); + kvm_x86_ops->tlb_flush_guest (vcpu);
Quoted 5 lines
arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 2680
Quoted 1 line
arch/x86/kvm/x86.c +0
-0vcpu->arch.st.preempted =0 ;There is one additional place in
hyperv.cwhere we do TLB flush on behalf of the guest,kvm_hv_flush_tlb(). Currently, it doesKVM_REQ_TLB_FLUSH(resulting inkvm_x86_ops->tlb_flush()), do we need something likeKVM_REQ_TLB_FLUSH_GUESTinstead?
Yes, that would be better since INVEPT does not flush linear mappings. So, when EPT and VPID is enabled, KVM_REQ_TLB_FLUSH would not flush the guest's translations.
Paolo
Paolo Bonzini
maintainerRe: [PATCH 06/10] KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
In reply to Sean ChristophersonQuoted 10 lines · Sean Christopherson
Add a dedicated hook to handle flushing TLB entries on behalf of the guest, i.e. for a paravirtualized TLB flush, and use it directly instead of bouncing through
kvm_vcpu_flush_tlb(). Change the effective VMX implementation to never do INVEPT, i.e. to always flush via INVVPID. The INVEPT performed by__vmx_flush_tlb()when @invalidate_gpa=false and enable_vpid=0 is unnecessary, as it will only flushGPA->HPAmappings;GVA->GPAandGVA->HPAtranslations are flushed by VM-Enter when VPID is disabled, and changes in the guest pages tables only affect GVA->*PA mappings.
Quoted 10 lines
When EPT and VPID are enabled, doing INVVPID is not required (by Intel's architecture) to invalidate GPA mappings, i.e. TLB entries that cache
GPA->HPAtranslations can live across INVVPID asGPA->HPAmappings are associated with an EPTP, not a VPID. The intent of @invalidate_gpa is to informvmx_flush_tlb()that it needs to "invalidate gpa mappings", i.e. do INVEPT and not simply INVVPID. Other than nested VPID handling, which now callsvpid_sync_context()directly, the only scenario where KVM can safely do INVVPID instead of INVEPT (when EPT is enabled) is if KVM is flushing TLB entries from the guest's perspective, i.e. is invalidatingGLA->GPAmappings.
Since you need a v2, can you replace the name of mappings with "linear", "guest-physical" and "combined" as in the SDM? It takes a little to get used to them but it avoids three-letter acronym soup.
Paolo
PATCH 7/10 KVM: VMX: Clean up vmx_flush_tlb_gva()
In reply to Sean ChristophersonRefactor vmx_flush_tlb_gva() to remove a superfluous local variable and clean up its comment, which is oddly located below the code it is commenting.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/vmx.c +3 -7
@@ -2824,15 +2824,11 @@ static void exit_lmode(struct kvm_vcpu *vcpu)static void vmx_flush_tlb_gva (struct kvm_vcpu *vcpu,gva_t addr) { -int vpid =to_vmx (vcpu)->vpid; - -vpid_sync_vcpu_addr (vpid, addr); -/* -* If VPIDs are not supported or enabled, then the above is a no-op. -* But we don't really need a TLB flush in that case anyway, because -* each VM entry/exit includes an implicit flush when VPID is 0. +* vpid_sync_vcpu_addr() is a nop if vmx->vpid==0, see the comment in +* vmx_flush_tlb_guest() for an explanation of why this is ok. */ +vpid_sync_vcpu_addr (to_vmx (vcpu)->vpid, addr); }static void vmx_flush_tlb_guest (struct kvm_vcpu *vcpu)
Signature · 1 line
2.24.1
Re: [PATCH 07/10] KVM: VMX: Clean up vmx_flush_tlb_gva()
Quoted 19 lines · Sean Christopherson
Refactor
vmx_flush_tlb_gva()to remove a superfluous local variable and clean up its comment, which is oddly located below the code it is commenting.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)arch/x86/kvm/vmx/vmx.c +0
-0@@ -2824,15 +2824,11 @@ static void exit_lmode(struct kvm_vcpu *vcpu)static void vmx_flush_tlb_gva (struct kvm_vcpu *vcpu,gva_t addr) {
arch/x86/kvm/vmx/vmx.c static void exit_lmode(struct kvm_vcpu *vcpu) · line 2829
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2
-7-int vpid =to_vmx (vcpu)->vpid; - -vpid_sync_vcpu_addr (vpid, addr); -/* -* If VPIDs are not supported or enabled, then the above is a no-op. -* But we don't really need a TLB flush in that case anyway, because -* each VM entry/exit includes an implicit flush when VPID is 0. +* vpid_sync_vcpu_addr() is a nop if vmx->vpid==0, see the comment in +* vmx_flush_tlb_guest() for an explanation of why this is ok.
"OK" :-)
arch/x86/kvm/vmx/vmx.c static void exit_lmode(struct kvm_vcpu *vcpu) · line 2834
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1
-0*/ +vpid_sync_vcpu_addr (to_vmx (vcpu)->vpid, addr); }static void vmx_flush_tlb_guest (struct kvm_vcpu *vcpu)
Signature · 1 line
Vitaly
PATCH 8/10 KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()
In reply to Sean ChristophersonDrop @invalidate_gpa from ->tlb_flush() and kvm_vcpu_flush_tlb() now that all callers pass %true for said param.
Note, vmx_flush_tlb() now unconditionally passes %true to __vmx_flush_tlb(), the less straightforward VMX change will be handled in a future patch.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm.c | 10 +++++----- arch/x86/kvm/vmx/vmx.c | 4 ++-- arch/x86/kvm/vmx/vmx.h | 4 ++-- arch/x86/kvm/x86.c | 6 +++--- 6 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1095,7 +1095,7 @@ struct kvm_x86_ops {unsigned long (*get_rflags)(struct kvm_vcpu *vcpu);void (*set_rflags)(struct kvm_vcpu *vcpu,unsigned long rflags); -void (*tlb_flush)(struct kvm_vcpu *vcpu,bool invalidate_gpa); +void (*tlb_flush)(struct kvm_vcpu *vcpu);int (*tlb_remote_flush)(struct kvm *kvm);int (*tlb_remote_flush_with_range)(struct kvm *kvm,struct kvm_tlb_range *range);
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -5186,7 +5186,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)if (r) goto out;kvm_mmu_load_cr3 (vcpu); - kvm_x86_ops->tlb_flush (vcpu, true); + kvm_x86_ops->tlb_flush (vcpu); out:return r; }
arch/x86/kvm/svm.c +5 -5
@@ -385,7 +385,7 @@ module_param(dump_invalid_vmcb, bool, 0644);static u8 rsm_ins_bytes[] ="\x0f\xaa" ;static void svm_set_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0); -static void svm_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa); +static void svm_flush_tlb (struct kvm_vcpu *vcpu);static void svm_complete_interrupts (struct vcpu_svm *svm);static void svm_toggle_avic_for_irq_window (struct kvm_vcpu *vcpu,bool activate);static inline void avic_post_state_restore (struct kvm_vcpu *vcpu);
@@ -2634,7 +2634,7 @@ static int svm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)return 1 ;if (npt_enabled && ((old_cr4 ^ cr4) &X86_CR4_PGE )) -svm_flush_tlb (vcpu, true); +svm_flush_tlb (vcpu); vcpu->arch.cr4 = cr4;if (!npt_enabled)
@@ -3588,7 +3588,7 @@ static void enter_svm_guest_mode(struct vcpu_svm *svm, u64 vmcb_gpa, svm->nested.intercept_exceptions = nested_vmcb->control.intercept_exceptions; svm->nested.intercept = nested_vmcb->control.intercept; -svm_flush_tlb (&svm->vcpu, true); +svm_flush_tlb (&svm->vcpu); svm->vmcb->control.int_ctl = nested_vmcb->control.int_ctl |V_INTR_MASKING_MASK ;if (nested_vmcb->control.int_ctl &V_INTR_MASKING_MASK ) svm->vcpu.arch.hflags |=HF_VINTR_MASK ;
@@ -5591,7 +5591,7 @@ static int svm_set_identity_map_addr(struct kvm *kvm, u64 ident_addr)return 0 ; } -static void svm_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static void svm_flush_tlb (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -5610,7 +5610,7 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)static void svm_flush_tlb_guest (struct kvm_vcpu *vcpu) { -svm_flush_tlb (vcpu, true); +svm_flush_tlb (vcpu); }static void svm_prepare_guest_switch (struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/vmx.c +2 -2
@@ -6043,7 +6043,7 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu)if (flexpriority_enabled) { sec_exec_control |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; -vmx_flush_tlb (vcpu, true); +vmx_flush_tlb (vcpu); }break ;case LAPIC_MODE_X2APIC :
@@ -6061,7 +6061,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu, hpa_t hpa) {if (!is_guest_mode (vcpu)) {vmcs_write64 (APIC_ACCESS_ADDR , hpa); -vmx_flush_tlb (vcpu, true); +vmx_flush_tlb (vcpu); } }
arch/x86/kvm/vmx/vmx.h +2 -2
@@ -514,9 +514,9 @@ static inline void __vmx_flush_tlb(struct kvm_vcpu *vcpu, int vpid, } } -static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu) { -__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid, invalidate_gpa); +__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid, true); }static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx)
arch/x86/kvm/x86.c +3 -3
@@ -2646,10 +2646,10 @@ static void kvmclock_reset(struct kvm_vcpu *vcpu) vcpu->arch.time =0 ; } -static void kvm_vcpu_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static void kvm_vcpu_flush_tlb (struct kvm_vcpu *vcpu) { ++vcpu->stat.tlb_flush; - kvm_x86_ops->tlb_flush (vcpu, invalidate_gpa); + kvm_x86_ops->tlb_flush (vcpu); }static void record_steal_time (struct kvm_vcpu *vcpu)
@@ -8166,7 +8166,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_LOAD_CR3 , vcpu))kvm_mmu_load_cr3 (vcpu);if (kvm_check_request (KVM_REQ_TLB_FLUSH , vcpu)) -kvm_vcpu_flush_tlb (vcpu, true); +kvm_vcpu_flush_tlb (vcpu);if (kvm_check_request (KVM_REQ_REPORT_TPR_ACCESS , vcpu)) { vcpu->run->exit_reason =KVM_EXIT_TPR_ACCESS ; r =0 ;
Signature · 1 line
2.24.1
Re: [PATCH 08/10] KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()
Quoted 150 lines · Sean Christopherson
Drop @invalidate_gpa from ->tlb_flush() and
kvm_vcpu_flush_tlb()now that all callers pass %true for said param.Note,
vmx_flush_tlb()now unconditionally passes %true to__vmx_flush_tlb(), the less straightforward VMX change will be handled in a future patch.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm.c | 10 +++++----- arch/x86/kvm/vmx/vmx.c | 4 ++-- arch/x86/kvm/vmx/vmx.h | 4 ++-- arch/x86/kvm/x86.c | 6 +++--- 6 files changed, 14 insertions(+), 14 deletions(-)arch/x86/include/asm/kvm_host.h +1
-1@@ -1095,7 +1095,7 @@ struct kvm_x86_ops {unsigned long (*get_rflags)(struct kvm_vcpu *vcpu);void (*set_rflags)(struct kvm_vcpu *vcpu,unsigned long rflags); -void (*tlb_flush)(struct kvm_vcpu *vcpu,bool invalidate_gpa); +void (*tlb_flush)(struct kvm_vcpu *vcpu);int (*tlb_remote_flush)(struct kvm *kvm);int (*tlb_remote_flush_with_range)(struct kvm *kvm,struct kvm_tlb_range *range);arch/x86/kvm/mmu/mmu.c +1
-1@@ -5186,7 +5186,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)if (r) goto out;kvm_mmu_load_cr3 (vcpu); - kvm_x86_ops->tlb_flush (vcpu, true); + kvm_x86_ops->tlb_flush (vcpu); out:return r; }arch/x86/kvm/svm.c +5
-5@@ -385,7 +385,7 @@ module_param(dump_invalid_vmcb, bool, 0644);static u8 rsm_ins_bytes[] ="\x0f\xaa" ;static void svm_set_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0); -static void svm_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa); +static void svm_flush_tlb (struct kvm_vcpu *vcpu);static void svm_complete_interrupts (struct vcpu_svm *svm);static void svm_toggle_avic_for_irq_window (struct kvm_vcpu *vcpu,bool activate);static inline void avic_post_state_restore (struct kvm_vcpu *vcpu);@@ -2634,7 +2634,7 @@ static int svm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)return 1 ;if (npt_enabled && ((old_cr4 ^ cr4) &X86_CR4_PGE )) -svm_flush_tlb (vcpu, true); +svm_flush_tlb (vcpu); vcpu->arch.cr4 = cr4;if (!npt_enabled)@@ -3588,7 +3588,7 @@ static void enter_svm_guest_mode(struct vcpu_svm *svm, u64 vmcb_gpa, svm->nested.intercept_exceptions = nested_vmcb->control.intercept_exceptions; svm->nested.intercept = nested_vmcb->control.intercept; -svm_flush_tlb (&svm->vcpu, true); +svm_flush_tlb (&svm->vcpu); svm->vmcb->control.int_ctl = nested_vmcb->control.int_ctl |V_INTR_MASKING_MASK ;if (nested_vmcb->control.int_ctl &V_INTR_MASKING_MASK ) svm->vcpu.arch.hflags |=HF_VINTR_MASK ;@@ -5591,7 +5591,7 @@ static int svm_set_identity_map_addr(struct kvm *kvm, u64 ident_addr)return 0 ; } -static void svm_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static void svm_flush_tlb (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);@@ -5610,7 +5610,7 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)static void svm_flush_tlb_guest (struct kvm_vcpu *vcpu) { -svm_flush_tlb (vcpu, true); +svm_flush_tlb (vcpu); }static void svm_prepare_guest_switch (struct kvm_vcpu *vcpu)arch/x86/kvm/vmx/vmx.c +2
-2@@ -6043,7 +6043,7 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu)if (flexpriority_enabled) { sec_exec_control |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; -vmx_flush_tlb (vcpu, true); +vmx_flush_tlb (vcpu); }break ;case LAPIC_MODE_X2APIC :@@ -6061,7 +6061,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu, hpa_t hpa) {if (!is_guest_mode (vcpu)) {vmcs_write64 (APIC_ACCESS_ADDR , hpa); -vmx_flush_tlb (vcpu, true); +vmx_flush_tlb (vcpu); } }arch/x86/kvm/vmx/vmx.h +2
-2@@ -514,9 +514,9 @@ static inline void __vmx_flush_tlb(struct kvm_vcpu *vcpu, int vpid, } } -static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu) { -__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid, invalidate_gpa); +__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid, true); }static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx)arch/x86/kvm/x86.c +2
-2@@ -2646,10 +2646,10 @@ static void kvmclock_reset(struct kvm_vcpu *vcpu) vcpu->arch.time =0 ; } -static void kvm_vcpu_flush_tlb (struct kvm_vcpu *vcpu,bool invalidate_gpa) +static void kvm_vcpu_flush_tlb (struct kvm_vcpu *vcpu) { ++vcpu->stat.tlb_flush; - kvm_x86_ops->tlb_flush (vcpu, invalidate_gpa); + kvm_x86_ops->tlb_flush (vcpu); }static void record_steal_time (struct kvm_vcpu *vcpu)
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8172
Quoted 8 lines
arch/x86/kvm/x86.c +1
-1@@ -8166,7 +8166,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_LOAD_CR3 , vcpu))kvm_mmu_load_cr3 (vcpu);if (kvm_check_request (KVM_REQ_TLB_FLUSH , vcpu)) -kvm_vcpu_flush_tlb (vcpu, true); +kvm_vcpu_flush_tlb (vcpu);if (kvm_check_request (KVM_REQ_REPORT_TPR_ACCESS , vcpu)) { vcpu->run->exit_reason =KVM_EXIT_TPR_ACCESS ; r =0 ;
Signature · 1 line
Vitaly
PATCH 9/10 KVM: VMX: Drop @invalidate_gpa from __vmx_flush_tlb()
In reply to Sean ChristophersonDrop @invalidate_gpa from __vmx_flush_tlb() now that it's sole caller unconditionally passes %true, i.e. "(invalidate_gpa || !enable_vpid)" will always evaluate true.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.h | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/vmx.h +3 -4
@@ -501,10 +501,9 @@ static inline struct vmcs *alloc_vmcs(bool shadow)u64 construct_eptp (struct kvm_vcpu *vcpu,unsigned long root_hpa); -static inline void __vmx_flush_tlb (struct kvm_vcpu *vcpu,int vpid, -bool invalidate_gpa) +static inline void __vmx_flush_tlb (struct kvm_vcpu *vcpu,int vpid) { -if (enable_ept && (invalidate_gpa || !enable_vpid)) { +if (enable_ept) {if (!VALID_PAGE (vcpu->arch.mmu->root_hpa))return ; ept_sync_context(construct_eptp(vcpu,
@@ -516,7 +515,7 @@ static inline void __vmx_flush_tlb(struct kvm_vcpu *vcpu, int vpid,static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu) { -__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid, true); +__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid); }static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx)
Signature · 1 line
2.24.1
PATCH 10/10 KVM: VMX: Fold __vmx_flush_tlb() into vmx_flush_tlb()
In reply to Sean ChristophersonFold __vmx_flush_tlb() into its sole caller, vmx_flush_tlb(), now that all call sites that previously bounced through __vmx_flush_tlb() to force the INVVPID path instead call vpid_sync_context() directly.
Opportunistically add a comment to explain why INVEPT is necessary when EPT is enabled, even if VPID is disabled.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.h | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/vmx.h +9 -7
@@ -501,23 +501,25 @@ static inline struct vmcs *alloc_vmcs(bool shadow)u64 construct_eptp (struct kvm_vcpu *vcpu,unsigned long root_hpa); -static inline void __vmx_flush_tlb (struct kvm_vcpu *vcpu,int vpid) +static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu) { +/* +* INVEPT must be issued when EPT is enabled, irrespective of VPID, as +* the CPU is not required to invalidate GPA->HPA mappings on VM-Entry, +* even if VPID is disabled. GPA->HPA mappings are associated with the +* root EPT structure and not any particular VPID (INVVPID is also not +* required to invalidate GPA->HPA mappings). +*/ if (enable_ept) {if (!VALID_PAGE (vcpu->arch.mmu->root_hpa))return ;ept_sync_context (construct_eptp (vcpu, vcpu->arch.mmu->root_hpa)); }else { -vpid_sync_context (vpid); +vpid_sync_context (to_vmx (vcpu)->vpid); } } -static inline void vmx_flush_tlb (struct kvm_vcpu *vcpu) -{ -__vmx_flush_tlb (vcpu,to_vmx (vcpu)->vpid); -} -static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx) { vmx->current_tsc_ratio = vmx->vcpu.arch.tsc_scaling_ratio;
Signature · 1 line
2.24.1
Quoted 41 lines · Sean Christopherson
This series is technically x86 wide, but it only superficially affects SVM, the motivation and primary touchpoints are all about VMX.
The goal of this series to ultimately clean up
__vmx_flush_tlb(), which, for me, manages to be extremely confusing despite being only ten lines of code.The most confusing aspect of
__vmx_flush_tlb()is that it is overloaded for multiple uses:1) TLB flushes in response to a change in KVM's MMU
2) TLB flushes during nested VM-Enter/VM-Exit when VPID is enabled
3) Guest-scoped TLB flushes for paravirt TLB flushing
Handling (2) and (3) in the same flow as (1) is kludgy, because the rules for (1) are quite different than the rules for (2) and (3). They're all squeezed into
__vmx_flush_tlb()via the @invalidate_gpa param, which means "invalidate gpa mappings", not "invalidate a specific gpa"; it took me forever and a day to realize that.To clean things up, handle (2) by directly calling
vpid_sync_context()instead of bouncing through__vmx_flush_tlb(), and handle (3) via a dedicatedkvm_x86_opshook. This allows for a less tricky implementation ofvmx_flush_tlb()for (1), and (hopefully) clarifies the rules for what mappings must be invalidated when.Sean Christopherson (10):
KVM: VMX: Usevpid_sync_context()directly when possible
KVM: VMX: Movevpid_sync_vcpu_addr()down a few lines
KVM: VMX: Handle INVVPID fallback logic invpid_sync_vcpu_addr()
KVM: VMX: Fold vpid_sync_vcpu_{single,global}() into
vpid_sync_context()
KVM: nVMX: Usevpid_sync_vcpu_addr()to emulate INVVPID with address
KVM: x86: Move "flush guest's TLB" logic to separatekvm_x86_opshook
KVM: VMX: Clean upvmx_flush_tlb_gva()
KVM: x86: Drop @invalidate_gpa param fromkvm_x86_ops'tlb_flush()
KVM: VMX: Drop @invalidate_gpa from__vmx_flush_tlb()
KVM: VMX: Fold__vmx_flush_tlb()intovmx_flush_tlb()
Quoted 10 lines
arch/x86/include/asm/kvm_host.h | 8 +++++++- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm.c | 14 ++++++++++---- arch/x86/kvm/vmx/nested.c | 12 ++++-------- arch/x86/kvm/vmx/ops.h | 32 +++++++++----------------------- arch/x86/kvm/vmx/vmx.c | 26 +++++++++++++++++--------- arch/x86/kvm/vmx/vmx.h | 19 ++++++++++--------- arch/x86/kvm/x86.c | 8 ++++---- 8 files changed, 62 insertions(+), 59 deletions(-)