KVM: nVMX: A few TLB flushing fixes
Subsystems (4)
cover KVM: nVMX: A few TLB flushing fixes
Fix a couple of issues surfaced while working on TLB optimizations for nSVM. I did not add a Reported-by tag for Sashiko, because in both instances Sashiko did not report the problem (or the nSVM equivalent), but it spit out a false positive that lead to finding the problems (and I don't know how to capture that).
Sean Christopherson (1):
KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02
Yosry Ahmed (2):
KVM: nVMX: Always flush vpid02 on first use
KVM: nVM: Ensure INVVPID is emulated on the correct physical CPU
arch/x86/kvm/vmx/nested.c | 62 ++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 30 deletions(-)
base-commit: c1f7303302927f9cbf4efedf70f0512cde168c65
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
PATCH 1/3 KVM: nVMX: Always flush vpid02 on first use
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02. nested_vmx_transition_tlb_flush() will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Note that last_vpid is already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.
--- arch/x86/kvm/vmx/nested.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kvm/vmx/nested.c +10 -0
@@ -1290,6 +1290,9 @@ static void nested_vmx_transition_tlb_flush(struct kvm_vcpu *vcpu,* is the VPID incorporated into the MMU context. I.e. KVM must assume * that the new vpid12 has never been used and thus represents a new * guest ASID that cannot have entries in the TLB. +* +* Note, last_vpid is initialized as 0, so the first nested VM-Enter +* after VMXON will always flush the TLB to avoid using stale entries. */ if (is_vmenter && vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id;
@@ -5447,6 +5450,13 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) vmx->nested.vpid02 =allocate_vpid (); +/* +* Clear last_vpid to ensure that the VPID is flushed on the first +* nested VM-Enter. Otherwise, stale TLB entries from a previous life of +* the VPID (e.g. different vCPU or even different VM) could be used. +*/ + vmx->nested.last_vpid =0 ; + vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 10 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 8 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Signed-off-by Yosry Ahmed <[email protected]>---
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
In reply to Yosry AhmedQuoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.
vmcs12 :-)
Quoted 9 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.
Not sure whether it's better to set it to 0 in free_nested(), which also resets some other nested fields to clean slate AFAICT?
Quoted 3 lines
Signed-off-by Yosry Ahmed <[email protected]>
Anyway:
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 13 lines · Kai Huang
Quoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.vmcs12 :-)
Quoted 5 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 7 lines
Quoted 4 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Not sure whether it's better to set it to 0 in
free_nested(), which also resets some other nested fields to clean slate AFAICT?
It needs to be set on first use, for the same reason that kvm_mmu_load() flushes the root:
/*
* Flush any TLB entries for the new root, the provenance of the root
* is unknown. Even if KVM ensures there are no stale TLB entries
* for a freed root, in theory another hypervisor could have left
* stale entries. Flushing on alloc also allows KVM to skip the TLB
* flush when freeing a root (see kvm_tdp_mmu_put_root()). */
kvm_x86_call(flush_tlb_current)(vcpu);
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 25 lines · Sean Christopherson
Quoted 13 lines · Kai Huang
Quoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.vmcs12 :-)
Quoted 5 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 7 lines
Quoted 4 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Not sure whether it's better to set it to 0 in
free_nested(), which also resets some other nested fields to clean slate AFAICT?It needs to be set on first use, for the same reason that
kvm_mmu_load()flushes the root:
Quoted 8 lines
/*
* Flush any TLB entries for the new root, the provenance of the root
* is unknown. Even if KVM ensures there are no stale TLB entries
* for a freed root, in theory another hypervisor could have left
* stale entries. Flushing on alloc also allows KVM to skip the TLB
* flush when freeing a root (seekvm_tdp_mmu_put_root()). */
kvm_x86_call(flush_tlb_current)(vcpu);
I think you mean the "actual flush" needs to be done on the first use. But setting last_vpid to 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between setting last_vpid to 0 in enter_vmx_operation() and free_nested(), but maybe I am missing something.
But I guess doing it in enter_vmx_operation() matches the logic of "doing actual flush on first use" more :-)
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 43 lines · Kai
Quoted 25 lines · Sean Christopherson
Quoted 13 lines · Kai Huang
Quoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.vmcs12 :-)
Quoted 5 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 7 lines
Quoted 4 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Not sure whether it's better to set it to 0 in
free_nested(), which also resets some other nested fields to clean slate AFAICT?It needs to be set on first use, for the same reason that
kvm_mmu_load()flushes the root:Quoted 8 lines
/*
* Flush any TLB entries for the new root, the provenance of the root
* is unknown. Even if KVM ensures there are no stale TLB entries
* for a freed root, in theory another hypervisor could have left
* stale entries. Flushing on alloc also allows KVM to skip the TLB
* flush when freeing a root (seekvm_tdp_mmu_put_root()). */
kvm_x86_call(flush_tlb_current)(vcpu);I think you mean the "actual flush" needs to be done on the first use. But setting
last_vpidto 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between settinglast_vpidto 0 inenter_vmx_operation()andfree_nested(), but maybe I am missing something.
Quoted 2 lines
But I guess doing it in
enter_vmx_operation()matches the logic of "doing actual flush on first use" more :-)
Yup. I thought about putting it free_nested() as it looks like cleanup, but semantically it makes more sense to put it in enter_vmx_operation().
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 44 lines · Yosry Ahmed
Quoted 43 lines · Kai
Quoted 25 lines · Sean Christopherson
Quoted 13 lines · Kai Huang
Quoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.vmcs12 :-)
Quoted 5 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 7 lines
Quoted 4 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Not sure whether it's better to set it to 0 in
free_nested(), which also resets some other nested fields to clean slate AFAICT?It needs to be set on first use, for the same reason that
kvm_mmu_load()flushes the root:Quoted 8 lines
/*
* Flush any TLB entries for the new root, the provenance of the root
* is unknown. Even if KVM ensures there are no stale TLB entries
* for a freed root, in theory another hypervisor could have left
* stale entries. Flushing on alloc also allows KVM to skip the TLB
* flush when freeing a root (seekvm_tdp_mmu_put_root()). */
kvm_x86_call(flush_tlb_current)(vcpu);I think you mean the "actual flush" needs to be done on the first use. But setting
last_vpidto 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between settinglast_vpidto 0 inenter_vmx_operation()andfree_nested(), but maybe I am missing something.
Quoted 6 lines
Quoted 2 lines
But I guess doing it in
enter_vmx_operation()matches the logic of "doing actual flush on first use" more :-)Yup. I thought about putting it
free_nested()as it looks like cleanup, but semantically it makes more sense to put it inenter_vmx_operation().
Sounds good to me. :-)
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I think you mean the "actual flush" needs to be done on the first use. But setting
last_vpidto 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between settinglast_vpidto 0 inenter_vmx_operation()andfree_nested(), but maybe I am missing something.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
But I guess doing it in
enter_vmx_operation()matches the logic of "doing actual flush on first use" more :-)Yup. I thought about putting it
free_nested()as it looks like cleanup, but semantically it makes more sense to put it inenter_vmx_operation().Sounds good to me. :-)
Thanks for taking a look and reviewing!
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
In reply to Yosry AhmedQuoted 37 lines · Yosry Ahmed
Quoted 36 lines · Kai
Quoted 33 lines · Sean Christopherson
Quoted 13 lines · Kai Huang
Quoted 4 lines · Yosry Ahmed
Make sure vpid02 is always flushed on first use by setting last_vpid=0 when allocating vpid02.
nested_vmx_transition_tlb_flush()will always detect a VPID change on first VM-Enter after VMXON, because VPID=0 in vmcb12 is not allowed if L1 enables VPID.vmcs12 :-)
Quoted 5 lines
This avoids using stale TLB entries from a previous lifetime of the VPID, that might have been associated with a different vCPU (or a completely different VM).
Quoted 7 lines
Quoted 4 lines
Note that
last_vpidis already being initialized as 0 when the vCPU is created, but it is not reset when vpid02 is freed on VMXOFF. Hence, the problem can only occur if L1 does VMXOFF -> VMXON, runs an L2, and KVM happens to reuse a VPID that has TLB entries on the physical CPU.Not sure whether it's better to set it to 0 in
free_nested(), which also resets some other nested fields to clean slate AFAICT?It needs to be set on first use, for the same reason that
kvm_mmu_load()flushes the root:/*
* Flush any TLB entries for the new root, the provenance of the root
* is unknown. Even if KVM ensures there are no stale TLB entries
* for a freed root, in theory another hypervisor could have left
* stale entries. Flushing on alloc also allows KVM to skip the TLB
* flush when freeing a root (seekvm_tdp_mmu_put_root()). */
kvm_x86_call(flush_tlb_current)(vcpu);
Quoted 6 lines
Quoted 6 lines
I think you mean the "actual flush" needs to be done on the first use. But setting
last_vpidto 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between settinglast_vpidto 0 inenter_vmx_operation()andfree_nested(), but maybe I am missing something.
You're not missing anything. It's just that putting it in free_nested() subtly relies on zero-allocating vmx->nested, so that the *very* first use also flushes vpid02. Relying on zero-allocating is generally a-ok, but in this case it would require documenting the same base logic in multiple places.
Quoted 6 lines
Quoted 2 lines
But I guess doing it in
enter_vmx_operation()matches the logic of "doing actual flush on first use" more :-)Yup. I thought about putting it
free_nested()as it looks like cleanup, but semantically it makes more sense to put it inenter_vmx_operation().
Re: [PATCH 1/3] KVM: nVMX: Always flush vpid02 on first use
Quoted 11 lines · Sean Christopherson
Quoted 6 lines
Quoted 6 lines
I think you mean the "actual flush" needs to be done on the first use. But setting
last_vpidto 0 is a setting which is to make sure the actual flush will always be done on the first use, i.e., the actual flush will always be done on the first use. For this purpose seems to me there's no difference between settinglast_vpidto 0 inenter_vmx_operation()andfree_nested(), but maybe I am missing something.You're not missing anything. It's just that putting it in
free_nested()subtly relies on zero-allocatingvmx->nested, so that the *very* first use also flushes vpid02. Relying on zero-allocating is generally a-ok, but in this case it would require documenting the same base logic in multiple places.
Right. Agreed.
PATCH 2/3 KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02
In reply to Yosry AhmedFrom: Sean Christopherson <[email protected]>
Separate the INVVPID operand checks from the actual flushing of vpid02 so the flushing can be adjusted to do the right thing when vmcs12 was last loaded on a different pCPU, without having to duplicate the logic across multiple case-statements.
Opportunistically let the VM-Fail paths poke out past 80 chars.
No functional change intended.
--- arch/x86/kvm/vmx/nested.c | 43 ++++++++++++--------------------------- 1 file changed, 13 insertions(+), 30 deletions(-)
arch/x86/kvm/vmx/nested.c +13 -30
@@ -6084,7 +6084,6 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)u64 vpid;u64 gla; } operand; -u16 vpid02;int r, gpr_index; if (!(vmx->nested.msrs.secondary_ctls_high &
@@ -6119,8 +6118,15 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)return kvm_handle_memory_failure (vcpu, r, &e);if (operand.vpid >>16 ) -return nested_vmx_fail (vcpu, -VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); +return nested_vmx_fail (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); + +if (type !=VMX_VPID_EXTENT_ALL_CONTEXT && !operand.vpid) +return nested_vmx_fail (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); + +/* LAM doesn't apply to addresses that are inputs to TLB invalidation. */ +if (type ==VMX_VPID_EXTENT_INDIVIDUAL_ADDR && +is_noncanonical_invlpg_address (operand.gla, vcpu)) +return nested_vmx_fail (vcpu,VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID );/* * Always flush the effective vpid02, i.e. never flush the current VPID
@@ -6128,33 +6134,10 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)* VMCS, and so whether or not the current vmcs12 has VPID enabled is * irrelevant (and there may not be a loaded vmcs12). */ - vpid02 =nested_get_vpid02 (vcpu); -switch (type) { -case VMX_VPID_EXTENT_INDIVIDUAL_ADDR : -/* -* LAM doesn't apply to addresses that are inputs to TLB -* invalidation. -*/ -if (!operand.vpid || -is_noncanonical_invlpg_address (operand.gla, vcpu)) -return nested_vmx_fail (vcpu, -VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); -vpid_sync_vcpu_addr (vpid02, operand.gla); -break ; -case VMX_VPID_EXTENT_SINGLE_CONTEXT : -case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL : -if (!operand.vpid) -return nested_vmx_fail (vcpu, -VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID ); -vpid_sync_context (vpid02); -break ; -case VMX_VPID_EXTENT_ALL_CONTEXT : -vpid_sync_context (vpid02); -break ; -default : -WARN_ON_ONCE (1 ); -return kvm_skip_emulated_instruction (vcpu); - } +if (type ==VMX_VPID_EXTENT_INDIVIDUAL_ADDR ) +vpid_sync_vcpu_addr (nested_get_vpid02 (vcpu), operand.gla); +else +vpid_sync_context (nested_get_vpid02 (vcpu));/* * Sync the shadow page tables if EPT is disabled, L1 is invalidating
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [PATCH 2/3] KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02
Quoted 4 lines · Yosry Ahmed
From: Sean Christopherson <[email protected]>
Separate the INVVPID operand checks from the actual flushing of vpid02 so the flushing can be adjusted to do the right thing when vmcs12 was last
Nit: vmcs12 or vmcs02?
Quoted 10 lines
loaded on a different pCPU, without having to duplicate the logic across multiple case-statements.
Opportunistically let the VM-Fail paths poke out past 80 chars.
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Yosry Ahmed <[email protected]>
Re: [PATCH 2/3] KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02
Quoted 8 lines · Kai
Quoted 4 lines · Yosry Ahmed
From: Sean Christopherson <[email protected]>
Separate the INVVPID operand checks from the actual flushing of vpid02 so the flushing can be adjusted to do the right thing when vmcs12 was last
Nit: vmcs12 or vmcs02?
I think vmcs02 is more accurate here, good eye.
Quoted 6 lines
Quoted 6 lines
loaded on a different pCPU, without having to duplicate the logic across multiple case-statements.
Opportunistically let the VM-Fail paths poke out past 80 chars.
Quoted 7 lines
Quoted 5 lines
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Yosry Ahmed <[email protected]>Reviewed-by Kai Huang <[email protected]>
Thank you!
PATCH 3/3 KVM: nVM: Ensure INVVPID is emulated on the correct physical CPU
In reply to Yosry AhmedWhen emulating INVVPID, KVM executes INVVPID on the physical CPU using vpid02 (instead of the L1 assigned VPID), after doing some validations on the operands. However, it is possible that the physical CPU KVM executes INVVPID on is different from the CPU L2 is running on.
For example, in the following scenario:
- L2 runs on CPU #1 and exits to L1 (vmx->nested.vmcs02.cpu=1)
- L1 migrates to CPU #2 and executes INVVPID
- KVM executes INVVPID on CPU #2
- L1 migrates back to CPU #1 and runs L2 (vmx->nested.vmcs02.cpu=1)
The TLB entries on CPU #1 are never invalidated, because INVVPID was executed on CPU #2, and vmcs02 never ran on a different pCPU (i.e. vmx_vcpu_load_vmcs() will *not* request KVM_REQ_TLB_FLUSH).
Ensure that INVVPID is being executed on the same pCPU that L2 last ran on, and if not, fallback to clearing last_vpid=0 to trigger a full VPID flush on the next nested VM-Enter (as KVM will detect L1 using a different VPID for L2). If L2 ends up running on a different pCPU, KVM will flush the TLB anyway through vmx_vcpu_load_vmcs().
--- arch/x86/kvm/vmx/nested.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/nested.c +10 -1
@@ -6085,6 +6085,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)u64 gla; } operand;int r, gpr_index; +int cpu; if (!(vmx->nested.msrs.secondary_ctls_high & SECONDARY_EXEC_ENABLE_VPID) ||
@@ -6133,11 +6134,19 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)* and never explicitly flush vpid01. INVVPID targets a VPID, not a * VMCS, and so whether or not the current vmcs12 has VPID enabled is * irrelevant (and there may not be a loaded vmcs12). +* +* If vmcs02 was last loaded on a different pCPU, then defer the flush +* by invalidating the nested VPID tracking to ensure that KVM performs +* the invalidation on the correct pCPU. */ -if (type ==VMX_VPID_EXTENT_INDIVIDUAL_ADDR ) + cpu =get_cpu (); +if (cpu != vmx->nested.vmcs02.cpu) + vmx->nested.last_vpid =0 ; +else if (type ==VMX_VPID_EXTENT_INDIVIDUAL_ADDR )vpid_sync_vcpu_addr (nested_get_vpid02 (vcpu), operand.gla);else vpid_sync_context (nested_get_vpid02 (vcpu)); +put_cpu ();/* * Sync the shadow page tables if EPT is disabled, L1 is invalidating
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [PATCH 3/3] KVM: nVM: Ensure INVVPID is emulated on the correct physical CPU
Quoted 15 lines · Yosry Ahmed
When emulating INVVPID, KVM executes INVVPID on the physical CPU using vpid02 (instead of the L1 assigned VPID), after doing some validations on the operands. However, it is possible that the physical CPU KVM executes INVVPID on is different from the CPU L2 is running on.
For example, in the following scenario:
- L2 runs on CPU #1 and exits to L1 (vmx->nested.vmcs02.cpu=1)
- L1 migrates to CPU #2 and executes INVVPID
- KVM executes INVVPID on CPU #2
- L1 migrates back to CPU #1 and runs L2 (vmx->nested.vmcs02.cpu=1)The TLB entries on CPU #1 are never invalidated, because INVVPID was executed on CPU #2, and vmcs02 never ran on a different pCPU (i.e.
vmx_vcpu_load_vmcs()will *not* requestKVM_REQ_TLB_FLUSH).
Quoted 9 lines
Ensure that INVVPID is being executed on the same pCPU that L2 last ran on, and if not, fallback to clearing last_vpid=0 to trigger a full VPID flush on the next nested VM-Enter (as KVM will detect L1 using a different VPID for L2). If L2 ends up running on a different pCPU, KVM will flush the TLB anyway through
vmx_vcpu_load_vmcs().Signed-off-by Yosry Ahmed <[email protected]>
Quoted 10 lines · Yosry Ahmed
Fix a couple of issues surfaced while working on TLB optimizations for nSVM. I did not add a Reported-by tag for Sashiko, because in both instances Sashiko did not report the problem (or the nSVM equivalent), but it spit out a false positive that lead to finding the problems (and I don't know how to capture that).
Sean Christopherson (1):
KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02[...]
Applied to kvm-x86 misc, thanks!
[1/3] KVM: nVMX: Always flush vpid02 on first use
https://github.com/kvm-x86/linux/commit/f0772389413d [2/3] KVM: nVMX: Decouple INVVPID operand checks from flushing of vpid02 https://github.com/kvm-x86/linux/commit/32912404b4b1 [3/3] KVM: nVM: Ensure INVVPID is emulated on the correct physical CPU https://github.com/kvm-x86/linux/commit/6d00e67326d8