KVM: x86: do not report a vCPU as preempted outside instruction boundaries
Subsystems (3)
Sasha Levin
maintainerPATCH 1/4 KVM: x86: do not report a vCPU as preempted outside instruction boundaries
From: Paolo Bonzini <[email protected]>
[ Upstream commit 6cd88243c7e03845a450795e134b488fc2afb736 ]
If a vCPU is outside guest mode and is scheduled out, it might be in the process of making a memory access. A problem occurs if another vCPU uses the PV TLB flush feature during the period when the vCPU is scheduled out, and a virtual address has already been translated but has not yet been accessed, because this is equivalent to using a stale TLB entry.
To avoid this, only report a vCPU as preempted if sure that the guest is at an instruction boundary. A rescheduling request will be delivered to the host physical CPU as an external interrupt, so for simplicity consider any vmexit *not* instruction boundary except for external interrupts.
It would in principle be okay to report the vCPU as preempted also if it is sleeping in kvm_vcpu_block(): a TLB flush IPI will incur the vmentry/vmexit overhead unnecessarily, and optimistic spinning is also unlikely to succeed. However, leave it for later because right now kvm_vcpu_check_block() is doing memory accesses. Even though the TLB flush issue only applies to virtual memory address, it's very much preferrable to be conservative.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/x86.c | 22 ++++++++++++++++++++++ 4 files changed, 28 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -642,6 +642,7 @@ struct kvm_vcpu_arch {u64 ia32_misc_enable_msr;u64 smbase;u64 smi_count; +bool at_instruction_boundary;bool tpr_access_reporting;bool xsaves_enabled;u64 ia32_xss;
@@ -1271,6 +1272,8 @@ struct kvm_vcpu_stat {u64 nested_run;u64 directed_yield_attempted;u64 directed_yield_successful; +u64 preemption_reported; +u64 preemption_other;u64 guest_mode; };
arch/x86/kvm/svm/svm.c +2 -0
@@ -4263,6 +4263,8 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) { +if (to_svm (vcpu)->vmcb->control.exit_code ==SVM_EXIT_INTR ) + vcpu->arch.at_instruction_boundary = true; }static void svm_sched_in (struct kvm_vcpu *vcpu,int cpu)
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6390,6 +6390,7 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)return ;handle_interrupt_nmi_irqoff (vcpu,gate_offset (desc)); + vcpu->arch.at_instruction_boundary = true; }static void vmx_handle_exit_irqoff (struct kvm_vcpu *vcpu)
arch/x86/kvm/x86.c +22 -0
@@ -277,6 +277,8 @@ const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = {STATS_DESC_COUNTER (VCPU , nested_run),STATS_DESC_COUNTER (VCPU , directed_yield_attempted),STATS_DESC_COUNTER (VCPU , directed_yield_successful), +STATS_DESC_COUNTER (VCPU , preemption_reported), +STATS_DESC_COUNTER (VCPU , preemption_other), STATS_DESC_ICOUNTER(VCPU, guest_mode) };
@@ -4368,6 +4370,19 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)struct kvm_memslots *slots;static const u8 preempted =KVM_VCPU_PREEMPTED ; +/* +* The vCPU can be marked preempted if and only if the VM-Exit was on +* an instruction boundary and will not trigger guest emulation of any +* kind (see vcpu_run). Vendor specific code controls (conservatively) +* when this is true, for example allowing the vCPU to be marked +* preempted if and only if the VM-Exit was due to a host interrupt. +*/ +if (!vcpu->arch.at_instruction_boundary) { + vcpu->stat.preemption_other++; +return ; + } + + vcpu->stat.preemption_reported++;if (!(vcpu->arch.st.msr_val &KVM_MSR_ENABLED ))return ;
@@ -9936,6 +9951,13 @@ static int vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.l1tf_flush_l1d = true;for (;;) { +/* +* If another guest vCPU requests a PV TLB flush in the middle +* of instruction emulation, the rest of the emulation could +* use a stale page translation. Assume that any code after +* this point can start executing an instruction. +*/ + vcpu->arch.at_instruction_boundary = false;if (kvm_vcpu_running (vcpu)) { r =vcpu_enter_guest (vcpu); }else {
Signature · 1 line
2.35.1
Sasha Levin
maintainerPATCH 2/4 KVM: x86: do not set st->preempted when going back to user space
From: Paolo Bonzini <[email protected]>
[ Upstream commit 54aa83c90198e68eee8b0850c749bc70efb548da ]
Similar to the Xen path, only change the vCPU's reported state if the vCPU was actually preempted. The reason for KVM's behavior is that for example optimistic spinning might not be a good idea if the guest is doing repeated exits to userspace; however, it is confusing and unlikely to make a difference, because well-tuned guests will hardly ever exit KVM_RUN in the first place.
--- arch/x86/kvm/x86.c | 26 ++++++++++++++------------ arch/x86/kvm/xen.h | 6 ++++-- 2 files changed, 18 insertions(+), 14 deletions(-)
arch/x86/kvm/x86.c +14 -12
@@ -4412,19 +4412,21 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) {int idx; -if (vcpu->preempted && !vcpu->arch.guest_state_protected) - vcpu->arch.preempted_in_kernel = !static_call (kvm_x86_get_cpl)(vcpu); +if (vcpu->preempted) { +if (!vcpu->arch.guest_state_protected) + vcpu->arch.preempted_in_kernel = !static_call (kvm_x86_get_cpl)(vcpu); -/* -* Take the srcu lock as memslots will be accessed to check the gfn -* cache generation against the memslots generation. -*/ - idx =srcu_read_lock (&vcpu->kvm->srcu); -if (kvm_xen_msr_enabled (vcpu->kvm)) -kvm_xen_runstate_set_preempted (vcpu); -else -kvm_steal_time_set_preempted (vcpu); -srcu_read_unlock (&vcpu->kvm->srcu, idx); +/* +* Take the srcu lock as memslots will be accessed to check the gfn +* cache generation against the memslots generation. +*/ + idx =srcu_read_lock (&vcpu->kvm->srcu); +if (kvm_xen_msr_enabled (vcpu->kvm)) +kvm_xen_runstate_set_preempted (vcpu); +else +kvm_steal_time_set_preempted (vcpu); +srcu_read_unlock (&vcpu->kvm->srcu, idx); + }static_call (kvm_x86_vcpu_put)(vcpu); vcpu->arch.last_host_tsc =rdtsc ();
arch/x86/kvm/xen.h +4 -2
@@ -97,8 +97,10 @@ static inline void kvm_xen_runstate_set_preempted(struct kvm_vcpu *vcpu)* behalf of the vCPU. Only if the VMM does actually block * does it need to enter RUNSTATE_blocked. */ -if (vcpu->preempted) -kvm_xen_update_runstate_guest (vcpu, RUNSTATE_runnable); +if (WARN_ON_ONCE (!vcpu->preempted)) +return ; + +kvm_xen_update_runstate_guest (vcpu, RUNSTATE_runnable); }/* 32-bit compatibility definitions, also used natively in 32-bit build */
Signature · 1 line
2.35.1
Sasha Levin
maintainerPATCH 3/4 KVM: selftests: Make hyperv_clock selftest more stable
In reply to Sasha LevinFrom: Vitaly Kuznetsov <[email protected]>
[ Upstream commit eae260be3a0111a28fe95923e117a55dddec0384 ]
hyperv_clock doesn't always give a stable test result, especially with AMD CPUs. The test compares Hyper-V MSR clocksource (acquired either with rdmsr() from within the guest or KVM_GET_MSRS from the host) against rdtsc(). To increase the accuracy, increase the measured delay (done with nop loop) by two orders of magnitude and take the mean rdtsc() value before and after rdmsr()/KVM_GET_MSRS.
Message-Id: <[email protected]>
--- tools/testing/selftests/kvm/x86_64/hyperv_clock.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/x86_64/hyperv_clock.c +7 -3
@@ -44,7 +44,7 @@ static inline void nop_loop(void) {int i; -for (i =0 ; i <1000000 ; i++) +for (i =0 ; i <100000000 ; i++) asmvolatile ("nop" ); }
@@ -56,12 +56,14 @@ static inline void check_tsc_msr_rdtsc(void) tsc_freq =rdmsr (HV_X64_MSR_TSC_FREQUENCY );GUEST_ASSERT (tsc_freq >0 ); -/* First, check MSR-based clocksource */ +/* For increased accuracy, take mean rdtsc() before and afrer rdmsr() */ r1 =rdtsc (); t1 =rdmsr (HV_X64_MSR_TIME_REF_COUNT ); + r1 = (r1 +rdtsc ()) /2 ;nop_loop (); r2 =rdtsc (); t2 =rdmsr (HV_X64_MSR_TIME_REF_COUNT ); + r2 = (r2 +rdtsc ()) /2 ;GUEST_ASSERT (r2 > r1 && t2 > t1);
@@ -181,12 +183,14 @@ static void host_check_tsc_msr_rdtsc(struct kvm_vm *vm) tsc_freq =vcpu_get_msr (vm,VCPU_ID ,HV_X64_MSR_TSC_FREQUENCY );TEST_ASSERT (tsc_freq >0 ,"TSC frequency must be nonzero" ); -/* First, check MSR-based clocksource */ +/* For increased accuracy, take mean rdtsc() before and afrer ioctl */ r1 =rdtsc (); t1 =vcpu_get_msr (vm,VCPU_ID ,HV_X64_MSR_TIME_REF_COUNT ); + r1 = (r1 +rdtsc ()) /2 ;nop_loop (); r2 =rdtsc (); t2 =vcpu_get_msr (vm,VCPU_ID ,HV_X64_MSR_TIME_REF_COUNT ); + r2 = (r2 +rdtsc ()) /2 ;TEST_ASSERT (t2 > t1,"Time reference MSR is not monotonic (%ld <= %ld)" , t1, t2);
Signature · 1 line
2.35.1
Sasha Levin
maintainerPATCH 4/4 KVM: Don't null dereference ops->destroy
In reply to Sasha LevinFrom: Alexey Kardashevskiy <[email protected]>
[ Upstream commit e8bc2427018826e02add7b0ed0fc625a60390ae5 ]
A KVM device cleanup happens in either of two callbacks:
1) destroy() which is called when the VM is being destroyed;
2) release() which is called when a device fd is closed.
Most KVM devices use 1) but Book3s's interrupt controller KVM devices (XICS, XIVE, XIVE-native) use 2) as they need to close and reopen during the machine execution. The error handling in kvm_ioctl_create_device() assumes destroy() is always defined which leads to NULL dereference as discovered by Syzkaller.
This adds a checks for destroy!=NULL and adds a missing release().
This is not changing kvm_destroy_devices() as devices with defined release() should have been removed from the KVM devices list by then.
--- virt/kvm/kvm_main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c +4 -1
@@ -4166,8 +4166,11 @@ static int kvm_ioctl_create_device(struct kvm *kvm,kvm_put_kvm_no_destroy (kvm);mutex_lock (&kvm->lock);list_del (&dev->vm_node); +if (ops->release) + ops->release (dev);mutex_unlock (&kvm->lock); - ops->destroy (dev); +if (ops->destroy) + ops->destroy (dev);return ret; }
Signature · 1 line
2.35.1
Quoted 25 lines · Sasha Levin
From: Paolo Bonzini <[email protected]>
[ Upstream commit
6cd88243c7e03845a450795e134b488fc2afb736]If a vCPU is outside guest mode and is scheduled out, it might be in the process of making a memory access. A problem occurs if another vCPU uses the PV TLB flush feature during the period when the vCPU is scheduled out, and a virtual address has already been translated but has not yet been accessed, because this is equivalent to using a stale TLB entry.
To avoid this, only report a vCPU as preempted if sure that the guest is at an instruction boundary. A rescheduling request will be delivered to the host physical CPU as an external interrupt, so for simplicity consider any vmexit *not* instruction boundary except for external interrupts.
It would in principle be okay to report the vCPU as preempted also if it is sleeping in
kvm_vcpu_block(): a TLB flush IPI will incur the vmentry/vmexit overhead unnecessarily, and optimistic spinning is also unlikely to succeed. However, leave it for later because right nowkvm_vcpu_check_block()is doing memory accesses. Even though the TLB flush issue only applies to virtual memory address, it's very much preferrable to be conservative.
Quoted 3 lines
Reported-by Jann Horn <[email protected]>Signed-off-by Paolo Bonzini <[email protected]>Signed-off-by Sasha Levin <[email protected]>
This feature was introduced in commit f38a7b75267f1f (first in 4.16). I think the fix has to be applied all the way back to there (so additionally to what you already did, it'd have to be added to 4.19, 5.4 and 5.10)?
But it doesn't seem to apply cleanly to those older branches. Paolo, are you going to send stable backports of this?
Quoted 31 lines · Jann Horn
Quoted 25 lines · Sasha Levin
From: Paolo Bonzini <[email protected]>
[ Upstream commit
6cd88243c7e03845a450795e134b488fc2afb736]If a vCPU is outside guest mode and is scheduled out, it might be in the process of making a memory access. A problem occurs if another vCPU uses the PV TLB flush feature during the period when the vCPU is scheduled out, and a virtual address has already been translated but has not yet been accessed, because this is equivalent to using a stale TLB entry.
To avoid this, only report a vCPU as preempted if sure that the guest is at an instruction boundary. A rescheduling request will be delivered to the host physical CPU as an external interrupt, so for simplicity consider any vmexit *not* instruction boundary except for external interrupts.
It would in principle be okay to report the vCPU as preempted also if it is sleeping in
kvm_vcpu_block(): a TLB flush IPI will incur the vmentry/vmexit overhead unnecessarily, and optimistic spinning is also unlikely to succeed. However, leave it for later because right nowkvm_vcpu_check_block()is doing memory accesses. Even though the TLB flush issue only applies to virtual memory address, it's very much preferrable to be conservative.Quoted 3 lines
Reported-by Jann Horn <[email protected]>Signed-off-by Paolo Bonzini <[email protected]>Signed-off-by Sasha Levin <[email protected]>
Quoted 7 lines
This feature was introduced in commit
f38a7b75267f1f(first in 4.16). I think the fix has to be applied all the way back to there (so additionally to what you already did, it'd have to be added to 4.19, 5.4 and 5.10)?But it doesn't seem to apply cleanly to those older branches. Paolo, are you going to send stable backports of this?
Also, I think the same thing applies for "KVM: x86: do not set st->preempted when going back to user space"?